Virtual testing of autonomous vehicle control system

ABSTRACT

Methods and systems for assessing, detecting, and responding to malfunctions involving components of autonomous vehicles and/or smart homes are described herein. Autonomous operation features and related components can be assessed using direct or indirect data regarding operation to determine the robustness of autonomous systems, including the use of virtual assessment of software components within a simulated environment. A server may retrieve one or more routines associated with autonomous operation. The server may also generate a set of test data associated with test conditions. The server may also execute an emulator that virtually simulates autonomous environment. The test data may be presented to the routines executing in the emulator to generate output data. The server may then analyze the output data to determine a quality metric.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to and the benefit of the filing dateof the following applications: (1) provisional U.S. Patent ApplicationNo. 62/286,017 entitled “Autonomous Vehicle Routing, Maintenance, &Fault Determination,” filed on Jan. 22, 2016; (2) provisional U.S.Patent Application No. 62/287,659 entitled “Autonomous VehicleTechnology,” filed on Jan. 27, 2016; (3) provisional U.S. PatentApplication No. 62/302,990 entitled “Autonomous Vehicle Routing,” filedon Mar. 3, 2016; (4) provisional U.S. Patent Application No. 62/303,500entitled “Autonomous Vehicle Routing,” filed on Mar. 4, 2016; (5)provisional U.S. Patent Application No. 62/312,109 entitled “AutonomousVehicle Routing,” filed on Mar. 23, 2016; (6) provisional U.S. PatentApplication No. 62/349,884 entitled “Autonomous Vehicle Component andSystem Assessment,” filed on Jun. 14, 2016; (7) provisional U.S. PatentApplication No. 62/351,559 entitled “Autonomous Vehicle Component andSystem Assessment,” filed on Jun. 17, 2016; (8) provisional U.S. PatentApplication No. 62/373,084 entitled “Autonomous Vehicle Communications,”filed on Aug. 10, 2016; (9) provisional U.S. Patent Application No.62/376,044 entitled “Autonomous Operation Expansion through Caravans,”filed on Aug. 17, 2016; (10) provisional U.S. Patent Application No.62/380,686 entitled “Autonomous Operation Expansion through Caravans,”filed on Aug. 29, 2016; (11) provisional U.S. Patent Application No.62/381,848 entitled “System and Method for Autonomous Vehicle SharingUsing Facial Recognition,” filed on Aug. 31, 2016; (12) provisional U.S.Patent Application No. 62/406,595 entitled “Autonomous Vehicle ActionCommunications,” filed on Oct. 11, 2016; (13) provisional U.S. PatentApplication No. 62/406,600 entitled “Autonomous Vehicle PathCoordination,” filed on Oct. 11, 2016; (14) provisional U.S. PatentApplication No. 62/406,605 entitled “Autonomous Vehicle Signal Control,”filed on Oct. 11, 2016; (15) provisional U.S. Patent Application No.62/406,611 entitled “Autonomous Vehicle Application,” filed on Oct. 11,2016; (16) provisional U.S. Patent Application No. 62/415,668 entitled“Method and System for Enhancing the Functionality of a Vehicle,” filedon Nov. 1, 2016; (17) provisional U.S. Patent Application No. 62/415,672entitled “Method and System for Repairing a Malfunctioning AutonomousVehicle,” filed on Nov. 1, 2016; (18) provisional U.S. PatentApplication No. 62/415,673 entitled “System and Method for AutonomousVehicle Sharing Using Facial Recognition,” filed on Nov. 1, 2016; (19)provisional U.S. Patent Application No. 62/415,678 entitled “System andMethod for Autonomous Vehicle Ride Sharing Using Facial Recognition,”filed on Nov. 1, 2016; (20) provisional U.S. Patent Application No.62/418,988 entitled “Virtual Testing of Autonomous Vehicle ControlSystem,” filed on Nov. 8, 2016; (21) provisional U.S. Patent ApplicationNo. 62/418,999 entitled “Detecting and Responding to Autonomous VehicleCollisions,” filed on Nov. 8, 2016; (22) provisional U.S. PatentApplication No. 62/419,002 entitled “Automatic Repair on AutonomousVehicles,” filed on Nov. 8, 2016; (23) provisional U.S. PatentApplication No. 62/419,009 entitled “Autonomous Vehicle ComponentMalfunction Impact Assessment,” filed on Nov. 8, 2016; (24) provisionalU.S. Patent Application No. 62/419,017 entitled “Autonomous VehicleSensor Malfunction Detection,” filed on Nov. 8, 2016; (25) provisionalU.S. Patent Application No. 62/419,023 entitled “Autonomous VehicleDamage and Salvage Assessment,” filed on Nov. 8, 2016; (26) provisionalU.S. Patent Application No. 62/424,078 entitled “Systems and Methods forSensor Monitoring,” filed Nov. 18, 2016; (27) provisional U.S. PatentApplication No. 62/424,093 entitled “Autonomous Vehicle SensorMalfunction Detection,” filed on Nov. 18, 2016; (28) provisional U.S.Patent Application No. 62/428,843 entitled “Autonomous Vehicle Control,”filed on Dec. 1, 2016; (29) provisional U.S. Patent Application No.62/430,215 entitled Autonomous Vehicle Environment and ComponentMonitoring,” filed on Dec. 5, 2016; (30) provisional U.S. PatentApplication No. 62/434,355 entitled “Virtual Testing of AutonomousEnvironment Control System,” filed Dec. 14, 2016; (31) provisional U.S.Patent Application No. 62/434,359 entitled “Detecting and Responding toAutonomous Environment Incidents,” filed Dec. 14, 2016; (32) provisionalU.S. Patent Application No. 62/434,361 entitled “Component Damage andSalvage Assessment,” filed Dec. 14, 2016; (33) provisional U.S. PatentApplication No. 62/434,365 entitled “Sensor Malfunction Detection,”filed Dec. 14, 2016; (34) provisional U.S. Patent Application No.62/434,368 entitled “Component Malfunction Impact Assessment,” filedDec. 14, 2016; and (35) provisional U.S. Patent Application No.62/434,370 entitled “Automatic Repair of Autonomous Components,” filedDec. 14, 2016. The entire contents of each of the preceding applicationsare hereby expressly incorporated herein by reference.

Additionally, the present application is related to the followingco-pending U.S. patent applications: (1) U.S. patent application Ser.No. ______ entitled “Autonomous Operation Suitability Assessment andMapping,” filed ______; (2) U.S. patent application Ser. No. ______entitled “Autonomous Vehicle Routing,” filed ______; (3) U.S. patentapplication Ser. No. ______ entitled “Autonomous Vehicle Routing DuringEmergencies,” filed ______; (4) U.S. patent application Ser. No. ______entitled “Autonomous Vehicle Trip Routing,” filed ______; (5) U.S.patent application Ser. No. ______ entitled “Autonomous VehicleParking,” filed ______; (6) U.S. patent application Ser. No. ______entitled “Autonomous Vehicle Retrieval,” filed ______; (7) U.S. patentapplication Ser. No. ______ entitled “Autonomous Vehicle ActionCommunications,” filed ______; (8) U.S. patent application Ser. No.______ entitled “Autonomous Vehicle Path Coordination,” filed ______;(9) U.S. patent application Ser. No. ______ entitled “Autonomous VehicleSignal Control,” filed ______; (10) U.S. patent application Ser. No.______ entitled “Autonomous Vehicle application,” filed ______; (11)U.S. patent application Ser. No. ______ entitled “Method and System forEnhancing the Functionality of a Vehicle,” filed ______; (12) U.S.patent application Ser. No. ______ entitled “Method and System forRepairing a Malfunctioning Autonomous Vehicle,” filed ______; (13) U.S.patent application Ser. No. ______ entitled “System and Method forAutonomous Vehicle Sharing Using Facial Recognition,” filed ______; (14)U.S. patent application Ser. No. ______ entitled “System and Method forAutonomous Vehicle Ride Sharing Using Facial Recognition,” filed ______;(15) U.S. patent application Ser. No. ______ entitled “AutonomousVehicle Sensor Malfunction Detection,” filed ______; (16) U.S. patentapplication Ser. No. ______ entitled “Sensor Malfunction Detection,”filed ______; (17) U.S. patent application Ser. No. ______ entitled“Autonomous Vehicle Component Malfunction Impact Assessment,” filed______; (18) U.S. patent application Ser. No. ______ entitled “ComponentMalfunction Impact Assessment,” filed ______; (19) U.S. patentapplication Ser. No. ______ entitled “Automatic Repair of AutonomousVehicles,” filed ______; (20) U.S. patent application Ser. No. ______entitled “Automatic Repair of Autonomous Components,” filed ______; (21)U.S. patent application Ser. No. ______ entitled “Autonomous VehicleDamage and Salvage Assessment,” filed ______; (22) U.S. patentapplication Ser. No. ______ entitled “Component Damage and SalvageAssessment,” filed ______; (23) U.S. patent application Ser. No. ______entitled “Detecting and Responding to Autonomous Vehicle Collisions,”filed ______; (24) U.S. patent application Ser. No. ______ entitled“Detecting and Responding to Autonomous Environment Incidents,” filed______; (25) U.S. patent application Ser. No. ______ entitled “VirtualTesting of Autonomous Vehicle Control System,” filed ______; (26) U.S.patent application Ser. No. ______ entitled “Virtual Testing ofAutonomous Environment Control System,” filed ______; (27) U.S. patentapplication Ser. No. ______ entitled “Autonomous Electric VehicleCharging,” filed ______; (28) U.S. patent application Ser. No. ______entitled “Coordinated Autonomous Vehicle Automatic Area Scanning,” filed______; (29) U.S. patent application Ser. No. ______ entitled“Operator-Specific Configuration of Autonomous Vehicle Operation,” filed______; (30) U.S. patent application Ser. No. ______ entitled“Autonomous Vehicle Operation Adjustment Based Upon Route,” filed______; (31) U.S. patent application Ser. No. ______ entitled“Autonomous Vehicle Component Maintenance and Repair,” filed ______; and(32) U.S. patent application Ser. No. ______ entitled “AnomalousCondition Detection and Response for Autonomous Vehicles,” filed ______.

FIELD

The present disclosure generally relates to systems and methods forcomponent monitoring and assessment for damage or other malfunctions.

BACKGROUND

Vehicles are typically operated by a human vehicle operator who controlsboth steering and motive controls. Operator error, inattention,inexperience, misuse, or distraction leads to many vehicle collisionseach year, resulting in injury and damage. Autonomous or semi-autonomousvehicles augment vehicle operators' information or replace vehicleoperators' control commands to operate the vehicle, in whole or part,with computer systems based upon information from sensors within, orattached to, the vehicle. Such vehicles may be operated with or withoutpassengers, thus requiring different means of control than traditionalvehicles. Such vehicles also may include a plurality of advancedsensors, capable of providing significantly more data (both in type andquantity) than is available even from GPS navigation assistance systemsinstalled in traditional vehicles.

Ensuring safe operation of such autonomous or semi-autonomous vehiclesis of the utmost importance because the automated systems of thesevehicles may not function properly in all environments. Althoughautonomous operation may be safer than manual operation under ordinarydriving conditions, unusual or irregular environmental conditions maysignificantly impair the functioning of the autonomous operationfeatures controlling the autonomous vehicle. Under some conditions,autonomous operation may become impractical or excessively dangerous. Asan example, fog or heavy rain may greatly reduce the ability ofautonomous operation features to safely control the vehicle.Additionally, damage or other impairment of sensors or other componentsof autonomous systems may significantly increase the risks associatedwith autonomous operation. Such conditions may change frequently,thereby changing the safety of autonomous vehicle operation. Similarrisks associated with impaired sensors may also be present in a smarthome environment.

BRIEF SUMMARY

The present embodiments may be related to autonomous or semi-autonomousvehicle operation, including driverless operation of fully autonomousvehicles. The embodiments described herein relate particularly tovarious aspects of autonomous operation feature, component, and softwaremonitoring and/or assessment. When malfunctions or other problems aredetected, remedial responses may be determined and implemented.Alternatively, some aspects relate to assessment of features,components, or software, either generally or in particular situations.Specific systems and methods are summarized below. The methods andsystems summarized below may include additional, less, or alternateactions, including those discussed elsewhere herein.

In one aspect, a computer-implemented method for improving thefunctioning of a computer and/or evaluating autonomous vehicle controlsoftware may be provided. The method may include (1) retrieving, from amemory, computer-readable instructions comprising one or more routinesof an autonomous operation feature for controlling autonomous vehicles;(2) receiving a selection of a vehicle operating system; (3) generatinga set of test data for evaluating the autonomous operation feature, theset of test data including simulated sensor data associated with testconditions; (4) executing an emulator program configured to mimic theselected vehicle operating system running on an on-board computer of anautonomous vehicle; (5) implementing, within the emulator, the one ormore routines of the autonomous operation feature; (6) presenting thesimulated sensor data of the set of test data to the one or moreroutines; (7) recording output generated by the one or more routines inresponse to the simulated sensor data, wherein the output is associatedwith the simulated sensor data; and/or (8) generating a quality metricfor the autonomous operation feature based upon the recorded output. Themethod may include additional, less, or alternate actions, includingthose discussed elsewhere herein.

For instance, the simulated sensor data may include one or moresequences of simulated sensor signals associated with one or more timedurations of simulated vehicle operation. A portion of the simulatedsensor data presented to the one or more routines may be selected forpresentation based upon at least a portion of the output generated bythe one or more routines in response to another portion of the simulatedsensor data.

In some embodiments, generating the set of test data may includereceiving, via a user interface, a selection of one or more indicatorsof the test conditions, and retrieving, from a memory, the simulatedsensor data to form the set of test data based upon the one or moreindicators of the test conditions. In further embodiments, the set oftest data may further include simulated control data from one or moreadditional autonomous operation features. Such simulated control datamay be associated with the simulated sensor data, and the simulatedcontrol data may be presented to the one or more routines with thesimulated sensor data. The set of test data may likewise include aplurality of subsets of test data, each subset associated with acombination of environmental conditions to be simulated for evaluationof the autonomous operation feature.

The quality metric indicates one or more risk levels associated with theautonomous operation feature. Generating the quality metric may includecomparing the recorded output against baseline output values associatedwith the set of test data. Determining the quality metric may furtherinclude determining one or more measures of differences between therecorded output and the baseline output values, then determining thequality metric based upon the determined one or more measure ofdifferences. The baseline output values are associated with anotherversion of autonomous vehicle control software for the autonomousoperation feature. The determined differences may include one or moreindications of an improvement of the recorded output over the baselineoutput values.

In some embodiments, the emulator may operate at an accelerated speed topresent the one or more sequences of simulated sensor data to the one ormore routines in one or more actual time durations that are less thanthe one or more time durations of simulated vehicle operation.

In another aspect, a computer-implemented method for improving thefunctioning of a computer and/or evaluating smart home control softwaremay be provided. The method may include (1) retrieving, from a memory,computer-readable instructions comprising one or more routines of anautonomous operation feature for controlling smart homes; (2) receivinga selection of a smart home operating system; (3) generating a set oftest data for evaluating the autonomous operation feature, the set oftest data including simulated sensor data associated with testconditions; (4) executing an emulator program configured to mimic theselected smart home operating system running on a smart home controllerof a smart home; (5) implementing, within the emulator, the one or moreroutines of the autonomous operation feature; (6) presenting thesimulated sensor data of the set of test data to the one or moreroutines; (7) recording output generated by the one or more routines inresponse to the simulated sensor data, wherein the output is associatedwith the simulated sensor data; and/or (8) generating a quality metricfor the autonomous operation feature based upon the recorded output. Themethod may include additional, less, or alternate actions, includingthose discussed elsewhere herein.

For instance, the simulated sensor data may include one or moresequences of simulated sensor signals associated with one or more timedurations of simulated smart home operation. A portion of the simulatedsensor data presented to the one or more routines may be selected forpresentation based upon at least a portion of the output generated bythe one or more routines in response to another portion of the simulatedsensor data.

In some embodiments, generating the set of test data may includereceiving, via a user interface, a selection of one or more indicatorsof the test conditions, and retrieving, from a memory, the simulatedsensor data to form the set of test data based upon the one or moreindicators of the test conditions. In further embodiments, the set oftest data may further include simulated control data from one or moreadditional autonomous operation features. Such simulated control datamay be associated with the simulated sensor data, and the simulatedcontrol data may be presented to the one or more routines with thesimulated sensor data. The set of test data may likewise include aplurality of subsets of test data, each subset associated with acombination of environmental conditions to be simulated for evaluationof the autonomous operation feature.

The quality metric indicates one or more risk levels associated with theautonomous operation feature. Generating the quality metric may includecomparing the recorded output against baseline output values associatedwith the set of test data. Determining the quality metric may furtherinclude determining one or more measures of differences between therecorded output and the baseline output values, then determining thequality metric based upon the determined one or more measure ofdifferences. The baseline output values are associated with anotherversion of smart home control software for the autonomous operationfeature. The determined differences may include one or more indicationsof an improvement of the recorded output over the baseline outputvalues.

In some embodiments, the emulator may operate at an accelerated speed topresent the one or more sequences of simulated sensor data to the one ormore routines in one or more actual time durations that are less thanthe one or more time durations of simulated smart home operation.

Systems or computer-readable media storing instructions for implementingall or part of the methods described above may also be provided in someaspects. Systems for implementing such methods may include one or moreof the following: a special-purpose assessment computing device, amobile computing device, an on-board computer, a remote server, one ormore sensors, one or more communication modules configured tocommunicate wirelessly via radio links, radio frequency links, and/orwireless communication channels, and/or one or more program memoriescoupled to one or more processors of the mobile computing device,on-board computer, or remote server. Such program memories may storeinstructions to cause the one or more processors to implement part orall of the method described above. Additional or alternative featuresdescribed herein below may be included in some aspects.

BRIEF DESCRIPTION OF THE DRAWINGS

Advantages will become more apparent to those skilled in the art fromthe following description of the preferred embodiments which have beenshown and described by way of illustration. As will be realized, thepresent embodiments may be capable of other and different embodiments,and their details are capable of modification in various respects.Accordingly, the drawings and description are to be regarded asillustrative in nature and not as restrictive.

The figures described below depict various aspects of the applications,methods, and systems disclosed herein. It should be understood that eachfigure depicts an embodiment of a particular aspect of the disclosedapplications, systems and methods, and that each of the figures isintended to accord with a possible embodiment thereof. Furthermore,wherever possible, the following description refers to the referencenumerals included in the following figures, in which features depictedin multiple figures are designated with consistent reference numerals.

FIG. 1A illustrates a block diagram of an exemplary autonomous vehicledata system for autonomous vehicle operation, monitoring, and relatedfunctions;

FIG. 1B illustrates a block diagram of an exemplary autonomous vehiclemonitoring system, showing a plurality of vehicles and smartinfrastructure components;

FIG. 2 illustrates a block diagram of an exemplary on-board computer ormobile device;

FIG. 3 illustrates a flow diagram of an exemplary autonomous vehicleoperation computer-implemented method;

FIGS. 4A-B illustrate flow diagrams of exemplary autonomous operationmonitoring computer-implemented methods for obtaining and recordinginformation during autonomous operation;

FIG. 5 illustrates a flow diagram of an exemplary incident responsecomputer-implemented method for detecting and responding to incidentsinvolving a vehicle and/or smart home while engaged in fully autonomousoperation or in the absence of a vehicle operator and/or smart homeoccupant;

FIG. 6 illustrates a flow diagram of an exemplary salvage assessmentcomputer-implemented method for automatically determining damage and/orsalvage potential for components of an autonomous vehicle and/or a smarthome;

FIG. 7 illustrates a flow diagram of an exemplary malfunction detectioncomputer-implemented method for identifying malfunctioning sensors of anautonomous vehicle and/or a smart home;

FIG. 8 illustrates a flow diagram of an exemplary malfunction assessmentcomputer-implemented method for determining the impact of malfunctioningcomponents on the operation of an autonomous vehicle and/or a smarthome;

FIG. 9 illustrates a flow diagram of an exemplary autonomous repaircomputer-implemented method for identifying and repairing malfunctioningcomponents of an autonomous vehicle and/or a smart home; and

FIG. 10 illustrates a flow diagram of an exemplary autonomousenvironment control software evaluation computer-implemented method fortesting the quality of control decisions generated by autonomousoperation feature of an autonomous vehicle and/or smart home.

DETAILED DESCRIPTION

The systems and methods disclosed herein generally relate to variousaspects of autonomous operation feature, component, and softwaremonitoring and/or assessment. Responses to accidents, collisions, andother events causing malfunctions or damage are discussed below.Assessment of components and features may be performed as part ofdetecting malfunctions, determining repairs, determining componentoperating status, or generally evaluating effectiveness or reliabilityof components and features. To this end, the systems and methods mayinclude collecting, communicating, evaluating, predicting, and/orutilizing data associated with autonomous or semi-autonomous operationfeatures for controlling a vehicle. The autonomous operation featuresmay take full control of the vehicle under certain conditions, viz.fully autonomous operation, or the autonomous operation features mayassist the vehicle operator in operating the vehicle, viz. partiallyautonomous operation.

Fully autonomous operation features may include systems within thevehicle that pilot the vehicle to a destination with or without avehicle operator present (e.g., an operating system for a driverlesscar) and/or systems within smart homes capable of automaticallycontrolling smart equipment disposed therein. Partially autonomousoperation features may assist the vehicle operator in limited ways(e.g., automatic braking or collision avoidance systems). Fully orpartially autonomous operation features may perform specific functionsto control or assist in controlling some aspect of vehicle operation, orsuch features may manage or control other autonomous operation features.For example, a vehicle operating system may control numerous subsystemsthat each fully or partially control aspects of vehicle operation.

In addition to information regarding the position or movement of avehicle, autonomous operation features may collect and utilize otherinformation, such as data about other vehicles or control decisions ofthe vehicle. Such additional information may be used to improve vehicleoperation, route the vehicle to a destination, warn of componentmalfunctions, advise others of potential hazards, or for other purposesdescribed herein. Information may be collected, assessed, and/or sharedvia applications installed and executing on computing devices associatedwith various vehicles or vehicle operators, such as on-board computersof vehicles or smartphones of vehicle operators. By using computerapplications to obtain data, the additional information generated byautonomous features of autonomous vehicles and/or smart homes may beused to assess the autonomous features themselves while in operation orto provide pertinent information to non-autonomous vehicles and/or smarthomes through an electronic communication network. These and otheradvantages are further described below.

Autonomous operation features utilize data not available to a humanoperator, respond to conditions in the vehicle operating environmentfaster than human operators, and do not suffer fatigue or distraction.Thus, the autonomous operation features may also significantly affectvarious risks associated with operating a vehicle and/or a smart home.Alternatively, autonomous operation features may be incapable of someactions typically taken by human operators, particularly when thefeatures or other components of the vehicle are damaged or inoperableand/or a smart home is unoccupied. Moreover, combinations of autonomousoperation features may further affect operating risks due to synergiesor conflicts between features. To account for these effects on risk,some embodiments evaluate the quality of each autonomous operationfeature and/or combination of features. This may be accomplished bytesting the features and combinations in controlled environments, aswell as analyzing the effectiveness of the features in the ordinarycourse of vehicle operation. New autonomous operation features may beevaluated based upon controlled testing and/or estimatingordinary-course performance based upon data regarding other similarfeatures for which ordinary-course performance is known.

Some autonomous operation features may be adapted for use underparticular conditions, such as city driving or highway driving.Additionally, the vehicle operator may be able to configure settingsrelating to the features or may enable or disable the features at will.Therefore, some embodiments monitor use of the autonomous operationfeatures, which may include the settings or levels of feature use duringoperation. Information obtained by monitoring feature usage may be usedto determine risk levels associated with operation, either generally orin relation to a vehicle operator and/or smart home occupant. In suchsituations, total risk may be determined by a weighted combination ofthe risk levels associated with operation while autonomous operationfeatures are enabled (with relevant settings) and the risk levelsassociated with operation while autonomous operation features aredisabled. For fully autonomous vehicles, settings or configurationsrelating to vehicle operation may be monitored and used in determiningvehicle operating risk.

In some embodiments, information regarding the risks associated withoperation with and without the autonomous operation features may be usedto determine risk categories or premiums for a vehicle insurance policycovering a vehicle with autonomous operation features and/or for a homeinsurance policy covering a smart home with autonomous operationfeatures, as described elsewhere herein. Risk category or price may bedetermined based upon factors relating to the evaluated effectiveness ofthe autonomous features. The risk or price determination may alsoinclude traditional factors, such as location, vehicle type, and levelof vehicle use. For fully autonomous vehicles, factors relating tovehicle operators may be excluded entirely. For partially autonomousvehicles, factors relating to vehicle operators may be reduced inproportion to the evaluated effectiveness and monitored usage levels ofthe autonomous operation features. For vehicles with autonomouscommunication features that obtain information from external sources(e.g., other vehicles or infrastructure), the risk level and/or pricedetermination may also include an assessment of the availability ofexternal sources of information. Location and/or timing of vehicle usemay thus be monitored and/or weighted to determine the risk associatedwith operation of the vehicle.

Exemplary Autonomous Vehicle Operation System

FIG. 1A illustrates a block diagram of an exemplary autonomous vehicledata system 100 on which the exemplary methods described herein may beimplemented. The high-level architecture includes both hardware andsoftware applications, as well as various data communications channelsfor communicating data between the various hardware and softwarecomponents. The autonomous vehicle data system 100 may be roughlydivided into front-end components 102 and back-end components 104. Thefront-end components 102 may obtain information regarding a vehicle 108(e.g., a car, truck, motorcycle, etc.) and the surrounding environment.An on-board computer 114 may utilize this information to operate thevehicle 108 according to an autonomous operation feature or to assistthe vehicle operator in operating the vehicle 108. To monitor thevehicle 108, the front-end components 102 may include one or moresensors 120 installed within the vehicle 108 that may communicate withthe on-board computer 114. The front-end components 102 may furtherprocess the sensor data using the on-board computer 114 or a mobiledevice 110 (e.g., a smart phone, a tablet computer, a special purposecomputing device, smart watch, wearable electronics, etc.) to determinewhen the vehicle is in operation and information regarding the vehicle.

In some embodiments of the system 100, the front-end components 102 maycommunicate with the back-end components 104 via a network 130. Eitherthe on-board computer 114 or the mobile device 110 may communicate withthe back-end components 104 via the network 130 to allow the back-endcomponents 104 to record information regarding vehicle usage. Theback-end components 104 may use one or more servers 140 to receive datafrom the front-end components 102, store the received data, process thereceived data, and/or communicate information associated with thereceived or processed data.

The front-end components 102 may be disposed within or communicativelyconnected to one or more on-board computers 114, which may bepermanently or removably installed in the vehicle 108. The on-boardcomputer 114 may interface with the one or more sensors 120 within thevehicle 108 (e.g., a digital camera, a LIDAR sensor, an ultrasonicsensor, an infrared sensor, an ignition sensor, an odometer, a systemclock, a speedometer, a tachometer, an accelerometer, a gyroscope, acompass, a geolocation unit, radar unit, etc.), which sensors may alsobe incorporated within or connected to the on-board computer 114.

The front end components 102 may further include a communicationcomponent 122 to transmit information to and receive information fromexternal sources, including other vehicles, infrastructure, or theback-end components 104. In some embodiments, the mobile device 110 maysupplement the functions performed by the on-board computer 114described herein by, for example, sending or receiving information toand from the mobile server 140 via the network 130, such as over one ormore radio frequency links or wireless communication channels. In otherembodiments, the on-board computer 114 may perform all of the functionsof the mobile device 110 described herein, in which case no mobiledevice 110 may be present in the system 100.

Either or both of the mobile device 110 or on-board computer 114 maycommunicate with the network 130 over links 112 and 118, respectively.Either or both of the mobile device 110 or on-board computer 114 may runa Data Application for collecting, generating, processing, analyzing,transmitting, receiving, and/or acting upon data associated with thevehicle 108 (e.g., sensor data, autonomous operation feature settings,or control decisions made by the autonomous operation features) or thevehicle environment (e.g., other vehicles operating near the vehicle108). Additionally, the mobile device 110 and on-board computer 114 maycommunicate with one another directly over link 116.

The mobile device 110 may be either a general-use personal computer,cellular phone, smart phone, tablet computer, smart watch, wearableelectronics, or a dedicated vehicle monitoring or control device.Although only one mobile device 110 is illustrated, it should beunderstood that a plurality of mobile devices 110 may be used in someembodiments. The on-board computer 114 may be a general-use on-boardcomputer capable of performing many functions relating to vehicleoperation or a dedicated computer for autonomous vehicle operation.Further, the on-board computer 114 may be installed by the manufacturerof the vehicle 108 or as an aftermarket modification or addition to thevehicle 108. In some embodiments or under certain conditions, the mobiledevice 110 or on-board computer 114 may function as thin-client devicesthat outsource some or most of the processing to the server 140.

The sensors 120 may be removably or fixedly installed within the vehicle108 and may be disposed in various arrangements to provide informationto the autonomous operation features. Among the sensors 120 may beincluded one or more of a GPS unit, a radar unit, a LIDAR unit, anultrasonic sensor, an infrared sensor, an inductance sensor, a camera,an accelerometer, a tachometer, or a speedometer. Some of the sensors120 (e.g., radar, LIDAR, or camera units) may actively or passively scanthe vehicle environment for obstacles (e.g., other vehicles, buildings,pedestrians, etc.), roadways, lane markings, signs, or signals. Othersensors 120 (e.g., GPS, accelerometer, or tachometer units) may providedata for determining the location or movement of the vehicle 108. Othersensors 120 may be directed to the interior or passenger compartment ofthe vehicle 108, such as cameras, microphones, pressure sensors,thermometers, or similar sensors to monitor the vehicle operator and/orpassengers within the vehicle 108. Information generated or received bythe sensors 120 may be communicated to the on-board computer 114 or themobile device 110 for use in autonomous vehicle operation.

In further embodiments, the front-end components may include aninfrastructure communication device 124 for monitoring the status of oneor more infrastructure components 126. Infrastructure components 126 mayinclude roadways, bridges, traffic signals, gates, switches, crossings,parking lots or garages, toll booths, docks, hangars, or other similarphysical portions of a transportation system's infrastructure. Theinfrastructure communication device 124 may include or becommunicatively connected to one or more sensors (not shown) fordetecting information relating to the condition of the infrastructurecomponent 126. The sensors (not shown) may generate data relating toweather conditions, traffic conditions, or operating status of theinfrastructure component 126.

The infrastructure communication device 124 may be configured to receivethe sensor data generated and determine a condition of theinfrastructure component 126, such as weather conditions, roadintegrity, construction, traffic, available parking spaces, etc. Theinfrastructure communication device 124 may further be configured tocommunicate information to vehicles 108 via the communication component122. In some embodiments, the infrastructure communication device 124may receive information from one or more vehicles 108, while, in otherembodiments, the infrastructure communication device 124 may onlytransmit information to the vehicles 108. The infrastructurecommunication device 124 may be configured to monitor vehicles 108and/or communicate information to other vehicles 108 and/or to mobiledevices 110.

In some embodiments, the communication component 122 may receiveinformation from external sources, such as other vehicles orinfrastructure. The communication component 122 may also sendinformation regarding the vehicle 108 to external sources. To send andreceive information, the communication component 122 may include atransmitter and a receiver designed to operate according topredetermined specifications, such as the dedicated short-rangecommunication (DSRC) channel, wireless telephony, Wi-Fi, or otherexisting or later-developed communications protocols. The receivedinformation may supplement the data received from the sensors 120 toimplement the autonomous operation features. For example, thecommunication component 122 may receive information that an autonomousvehicle ahead of the vehicle 108 is reducing speed, allowing theadjustments in the autonomous operation of the vehicle 108.

In addition to receiving information from the sensors 120, the on-boardcomputer 114 may directly or indirectly control the operation of thevehicle 108 according to various autonomous operation features. Theautonomous operation features may include software applications ormodules implemented by the on-board computer 114 to generate andimplement control commands to control the steering, braking, or throttleof the vehicle 108. To facilitate such control, the on-board computer114 may be communicatively connected to control components of thevehicle 108 by various electrical or electromechanical controlcomponents (not shown). When a control command is generated by theon-board computer 114, it may thus be communicated to the controlcomponents of the vehicle 108 to effect a control action. In embodimentsinvolving fully autonomous vehicles, the vehicle 108 may be operableonly through such control components (not shown). In other embodiments,the control components may be disposed within or supplement othervehicle operator control components (not shown), such as steeringwheels, accelerator or brake pedals, or ignition switches.

In some embodiments, the front-end components 102 communicate with theback-end components 104 via the network 130. The network 130 may be aproprietary network, a secure public internet, a virtual private networkor some other type of network, such as dedicated access lines, plainordinary telephone lines, satellite links, cellular data networks,combinations of these. The network 130 may include one or more radiofrequency communication links, such as wireless communication links 112and 118 with mobile devices 110 and on-board computers 114,respectively. Where the network 130 comprises the Internet, datacommunications may take place over the network 130 via an Internetcommunication protocol.

The back-end components 104 include one or more servers 140. Each server140 may include one or more computer processors adapted and configuredto execute various software applications and components of theautonomous vehicle data system 100, in addition to other softwareapplications. The server 140 may further include a database 146, whichmay be adapted to store data related to the operation of the vehicle 108and/or a smart home (not depicted) and its autonomous operationfeatures. Such data might include, for example, dates and times ofvehicle use, duration of vehicle use, use and settings of autonomousoperation features, information regarding control decisions or controlcommands generated by the autonomous operation features, speed of thevehicle 108, RPM or other tachometer readings of the vehicle 108,lateral and longitudinal acceleration of the vehicle 108, vehicleaccidents, incidents or near collisions of the vehicle 108, hazardous oranomalous conditions within the vehicle operating environment (e.g.,construction, accidents, etc.), communication between the autonomousoperation features and external sources, environmental conditions ofvehicle operation (e.g., weather, traffic, road condition, etc.), errorsor failures of autonomous operation features, or other data relating touse of the vehicle 108 and the autonomous operation features, which maybe uploaded to the server 140 via the network 130. The server 140 mayaccess data stored in the database 146 when executing various functionsand tasks associated with the evaluating feature effectiveness orassessing risk relating to an autonomous vehicle.

Although the autonomous vehicle data system 100 is shown to include onevehicle 108, one mobile device 110, one on-board computer 114, and oneserver 140, it should be understood that different numbers of vehicles108, mobile devices 110, on-board computers 114, and/or servers 140 maybe utilized. For example, the system 100 may include a plurality ofservers 140 and hundreds or thousands of mobile devices 110 or on-boardcomputers 114, all of which may be interconnected via the network 130.Furthermore, the database storage or processing performed by the one ormore servers 140 may be distributed among a plurality of servers 140 inan arrangement known as “cloud computing.” This configuration mayprovide various advantages, such as enabling near real-time uploads anddownloads of information as well as periodic uploads and downloads ofinformation. This may in turn support a thin-client embodiment of themobile device 110 or on-board computer 114 discussed herein.

The server 140 may have a controller 155 that is operatively connectedto the database 146 via a link 156. It should be noted that, while notshown, additional databases may be linked to the controller 155 in aknown manner. For example, separate databases may be used for varioustypes of information, such as autonomous operation feature information,vehicle accidents, road conditions, vehicle insurance policyinformation, or vehicle use information. Additional databases (notshown) may be communicatively connected to the server 140 via thenetwork 130, such as databases maintained by third parties (e.g.,weather, construction, or road network databases). The controller 155may include a program memory 160, a processor 162 (which may be called amicrocontroller or a microprocessor), a random-access memory (RAM) 164,and an input/output (I/O) circuit 166, all of which may beinterconnected via an address/data bus 165. It should be appreciatedthat although only one microprocessor 162 is shown, the controller 155may include multiple microprocessors 162. Similarly, the memory of thecontroller 155 may include multiple RAMs 164 and multiple programmemories 160. Although the I/O circuit 166 is shown as a single block,it should be appreciated that the I/O circuit 166 may include a numberof different types of I/O circuits. The RAM 164 and program memories 160may be implemented as semiconductor memories, magnetically readablememories, or optically readable memories, for example. The controller155 may also be operatively connected to the network 130 via a link 135.

The server 140 may further include a number of software applicationsstored in a program memory 160. The various software applications on theserver 140 may include an autonomous operation information monitoringapplication 141 for receiving information regarding the vehicle 108 andits autonomous operation features (which may include control commands ordecisions of the autonomous operation features), a feature evaluationapplication 142 for determining the effectiveness of autonomousoperation features under various conditions and/or determining operatingcondition of autonomous operation features or components, a real-timecommunication application 143 for communicating information regardingvehicle or environmental conditions between a plurality of vehicles, anavigation application 144 for assisting autonomous or semi-autonomousvehicle operation, and an accident detection application 145 foridentifying accidents and providing assistance. The various softwareapplications may be executed on the same computer processor or ondifferent computer processors.

FIG. 1B illustrates a block diagram of an exemplary autonomous vehiclemonitoring system 180 on which the exemplary methods described hereinmay be implemented. In one aspect, system 180 may include a network 130,N number of vehicles 182.1-182.N and respective mobile computing devices184.1-184.N, an external computing device 186, N number of smart homes187.1-187.N, N number of personal electronic devices 189.1-189.N, and/ora smart infrastructure component 188. In one aspect, mobile computingdevices 184 may be an implementation of mobile computing device 110,while vehicles 182 may be an implementation of vehicle 108. The vehicles182 may include a plurality of vehicles 108 having autonomous operationfeatures, as well as a plurality of other vehicles not having autonomousoperation features. As illustrated, the vehicle 182.1 may include avehicle controller 181.1, which may be an on-board computer 114 asdiscussed elsewhere herein, while vehicle 182.2 may lack such acomponent. Each of vehicles 182.1 and 182.2 may be configured forwireless inter-vehicle communication, such as vehicle-to-vehicle (V2V)wireless communication and/or data transmission via the communicationcomponent 122, directly via the mobile computing devices 184, orotherwise.

As illustrated, the autonomous vehicle monitoring system 180 may monitordevices other than the vehicles 182. For instance, the autonomousvehicle monitoring system 180 may monitor the smart home 187.1. Thesmart home 187.1 may be associated with a smart home controller 185.1.Similar to how the vehicle controller 181.1 monitors a plurality ofsensors associated with the vehicle 182.1, the smart home controller185.1 may monitor a plurality of sensors associated with the smart home187.1. To this end, the smart home 187.1 may include a plurality ofsensors (not depicted) disposed on or proximate to the smart home 187.1.For example, the smart home 187.1 may include a smoke sensor, atemperature sensor, a flood level sensor, a motion sensor, an imagesensor, a thermal image sensor, and so on.

In embodiments, the smart home 187.1 may include a plurality of smartequipment (not depicted). The smart equipment may include appliances,electronics, electrical systems, gas systems, water systems, windows,doors, shutters, and so on configured to communicate with the smart homecontroller 185.1. The smart equipment may include one or more sensorsthat monitor the operation of the smart equipment. Additional detailsdescribing a smart home environment may be found in co-owned U.S. patentapplication Ser. No. 14/693,032 entitled “SYSTEMS AND METHODS FORAUTOMATICALLY MITIGATING RISK OF PROPERTY DAMAGE,” the entire disclosureof which is hereby incorporated by reference in its entirety. In anaspect, the smart home controller may communicate over the network 130via a communication link 183 g.

In an aspect, another example of a device other than the vehicles 182the autonomous vehicle monitoring system 180 may monitor includepersonal electronic device 189. The personal electronic devices 189 mayinclude any type of electronic device that monitors conditionsassociated with an individual. For example, the personal electronicdevice 189.1 may be a smart watch, a fitness tracker, a personal medicaldevice (e.g., a pace maker, an insulin pump, etc.) and/or monitoringdevices thereof, smart implants, and so on. The personal electronicdevice 189 may monitor the conditions of the individual while theindividual is present in the vehicle 182.1 and/or operating the vehicle182.1 in a semi-autonomous mode. In some embodiments, when the personalelectronic device 189.1 is within and/or proximate to the vehicle 182.1,the personal electronic device may be in communication with the vehiclecontroller 181.1 and/or the mobile computing device 184.1. Additionallyor alternatively, the personal electronic device 189.1 may communicateover the network 130 via a communication link 183 h.

Although system 180 is shown in FIG. 1B as including one network 130,two mobile computing devices 184.1 and 184.2, two vehicles 182.1 and182.2, one external computing device 186, one smart home 187.1, onepersonal electronic device 189.1, and/or one smart infrastructurecomponent 188, various embodiments of system 180 may include anysuitable number of networks 130, mobile computing devices 184, vehicles182, external computing devices 186, smart homes 187, personalelectronic devices 189 and/or infrastructure components 188. Thevehicles 182 included in such embodiments may include any number ofvehicles 182.i having vehicle controllers 181.i (such as vehicle 182.1with vehicle controller 181.1) and vehicles 182.j not having vehiclescontrollers (such as vehicle 182.2). Moreover, system 180 may include aplurality of external computing devices 186 and more than two mobilecomputing devices 184, any suitable number of which being interconnecteddirectly to one another and/or via network 130.

In one aspect, each of mobile computing devices 184.1 and 184.2 may beconfigured to communicate with one another directly via peer-to-peer(P2P) wireless communication and/or data transfer over a radio link orwireless communication channel. In other aspects, each of mobilecomputing devices 184.1 and 184.2 may be configured to communicateindirectly with one another and/or any suitable device viacommunications over network 130, such as external computing device 186,smart home controller 185.1, personal electronic device 189.1, and/orsmart infrastructure component 188, for example. In still other aspects,each of mobile computing devices 184.1 and 184.2 may be configured tocommunicate directly and/or indirectly with other suitable devices,which may include synchronous or asynchronous communication.

In one aspect, each of mobile computing devices 184.1 and 184.2 may beconfigured to communicate with one another, and/or with the personalelectronic device 189.1 and/or the smart home controller 185.1, directlyvia peer-to-peer (P2P) wireless communication and/or data transfer overa radio link or wireless communication channel. In other aspects, eachof mobile computing devices 184.1 and 184.2, and/or with the personalelectronic device 189.1 and/or the smart home controller 185.1, may beconfigured to communicate indirectly with one another and/or anysuitable device via communications over network 130, such as externalcomputing device 186 and/or smart infrastructure component 188, forexample. In still other aspects, each of mobile computing devices 184.1and 184.2, and/or with the personal electronic device 189.1 and/or thesmart home controller 185.1, may be configured to communicate directlyand/or indirectly with other suitable devices, which may includesynchronous or asynchronous communication.

Each of mobile computing devices 184.1 and 184.2, and/or with thepersonal electronic device 189.1 and/or the smart home controller 185.1,may be configured to send data to and/or receive data from one anotherand/or via network 130 using one or more suitable communicationprotocols, which may be the same communication protocols or differentcommunication protocols. For example, mobile computing devices 184.1 and184.2 may be configured to communicate with one another via a directradio link 183 a, which may utilize, for example, a Wi-Fi directprotocol, an ad-hoc cellular communication protocol, etc. Mobilecomputing devices 184.1 and 184.2 may also be configured to communicatewith vehicles 182.1 and 182.2, respectively, utilizing a Bluetoothcommunication protocol (radio link not shown). In some embodiments, thismay include communication between a mobile computing device 184.1 and avehicle controller 181.1. In other embodiments, it may involvecommunication between a mobile computing device 184.2 and a vehicletelephony, entertainment, navigation, or information system (not shown)of the vehicle 182.2 that provides functionality other than autonomous(or semi-autonomous) vehicle control. Thus, vehicles 182.2 withoutautonomous operation features may nonetheless be connected to mobilecomputing devices 184.2 in order to facilitate communication,information presentation, or similar non-control operations (e.g.,navigation display, hands-free telephony, or music selection andpresentation).

To provide additional examples, mobile computing devices 184.1 and 184.2may be configured to communicate with one another via radio links 183 band 183 c by each communicating with network 130 utilizing a cellularcommunication protocol. As an additional example, mobile computingdevices 184.1 and/or 184.2 may be configured to communicate withexternal computing device 186 via radio links 183 b, 183 c, and/or 183e. Still further, one or more of mobile computing devices 184.1 and/or184.2 may also be configured to communicate with one or more smartinfrastructure components 188 directly (e.g., via radio link 183 d)and/or indirectly (e.g., via radio links 183 c and 183 f via network130) using any suitable communication protocols. As yet another example,the one or more of mobile computing devices 184.1 and/or 184.2 may alsobe configured to communicate with one or more personal electronicdevices 189 directly (not depicted) and/or indirectly (e.g., via radiolinks 183 c and 183 h via network 130) using any suitable communicationprotocols. As still another example, the one or more of mobile computingdevices 184.1 and/or 184.2 may also be configured to communicate withone or more smart home controllers 185 directly (not depicted) and/orindirectly (e.g., via radio links 183 c and 183 g via network 130) usingany suitable communication protocols. Similarly, one or more vehiclecontrollers 181.1 may be configured to communicate directly to thenetwork 130 (via radio link 183 b) or indirectly through mobilecomputing device 184.1 (via radio link 183 b). Vehicle controllers 181.1may also communicate with other vehicle controllers and/or mobilecomputing devices 184.2 directly or indirectly through mobile computingdevice 184.1 via local radio links 183 a. As discussed elsewhere herein,network 130 may be implemented as a wireless telephony network (e.g.,GSM, CDMA, LTE, etc.), a Wi-Fi network (e.g., via one or more IEEE802.11 Standards), a WiMAX network, a Bluetooth network, etc. Thus,links 183 a-183 h may represent wired links, wireless links, or anysuitable combination thereof. For example, the links 183 e and/or 183 fmay include wired links to the network 130, in addition to, or insteadof, wireless radio connections.

In some embodiments, the external computing device 186 may mediatecommunication between the mobile computing devices 184.1 and 184.2,and/or the smart home controllers 185 and/or the personal electronicdevices 189, based upon location or other factors. In embodiments inwhich mobile computing devices 184.1 and 184.2 communicate directly withone another in a peer-to-peer fashion, network 130 may be bypassed andthus communications between mobile computing devices 184.1 and 184.2 andexternal computing device 186 may be unnecessary. For example, in someaspects, mobile computing device 184.1 may broadcast geographic locationdata and/or telematics data directly to mobile computing device 184.2.In this case, mobile computing device 184.2 may operate independently ofnetwork 130 to determine operating data, risks associated withoperation, control actions to be taken, and/or alerts to be generated atmobile computing device 184.2 based upon the geographic location data,sensor data, and/or the autonomous operation feature data. In accordancewith such aspects, network 130 and external computing device 186 may beomitted.

However, in other aspects, one or more of mobile computing devices 184.1and/or 184.2, and/or the smart home controllers 185 and/or the personalelectronic devices 189, may work in conjunction with external computingdevice 186 to determine operating data, risks associated with operation,control actions to be taken, and/or alerts to be generated. For example,in some aspects, mobile computing device 184.1 may broadcast geographiclocation data and/or autonomous operation feature data, which isreceived by external computing device 186. In this case, externalcomputing device 186 may be configured to determine whether the same orother information should be sent to mobile computing device 184.2 basedupon the geographic location data, autonomous operation feature data, ordata derived therefrom.

Mobile computing devices 184.1 and 184.2 may be configured to executeone or more algorithms, programs, applications, etc., to determine ageographic location of each respective mobile computing device (and thustheir associated vehicle) to generate, measure, monitor, and/or collectone or more sensor metrics as telematics data, to broadcast thegeographic data and/or telematics data via their respective radio links,to receive the geographic data and/or telematics data via theirrespective radio links, to determine whether an alert should begenerated based upon the telematics data and/or the geographic locationdata, to generate the one or more alerts, and/or to broadcast one ormore alert notifications. Such functionality may, in some embodiments becontrolled in whole or part by a Data Application operating on themobile computing devices 184, as discussed elsewhere herein. Such DataApplication may communicate between the mobile computing devices 184,the smart home controllers 185, the personal electronic devices 189, andone or more external computing devices 186 (such as servers 140) tofacilitate centralized data collection and/or processing.

In some embodiments, the Data Application may facilitate control of avehicle 182 by a user, such as by selecting vehicle destinations and/orroutes along which the vehicle 182 will travel. The Data Application mayfurther be used to establish restrictions on vehicle use or store userpreferences for vehicle use, such as in a user profile. In furtherembodiments, the Data Application may monitor vehicle operation orsensor data in real-time to make recommendations or for other purposesas described herein. The Data Application may further facilitatemonitoring and/or assessment of the vehicle 182, such as by evaluatingoperating data to determine the condition of the vehicle or componentsthereof (e.g., sensors, autonomous operation features, etc.).

External computing device 186 may be configured to execute varioussoftware applications, algorithms, and/or other suitable programs.External computing device 186 may be implemented as any suitable type ofdevice to facilitate the functionality as described herein. For example,external computing device 186 may be a server 140 as discuses elsewhereherein. As another example, the external computing device 186 may beanother computing device associated with an operator or owner of avehicle 182, such as a desktop or notebook computer. Althoughillustrated as a single device in FIG. 1B, one or more portions ofexternal computing device 186 may be implemented as one or more storagedevices that are physically co-located with external computing device186, or as one or more storage devices utilizing different storagelocations as a shared database structure (e.g. cloud storage).

In some embodiments, external computing device 186 may be configured toperform any suitable portion of the processing functions remotely thathave been outsourced by one or more of mobile computing devices 184.1and/or 184.2 (and/or vehicle controllers 181.1), and/or the smart homecontrollers 185 and/or the personal electronic devices 189. For example,mobile computing device 184.1 and/or 184.2 may collect data (e.g.,geographic location data and/or telematics data) as described herein,but may send the data to external computing device 186 for remoteprocessing instead of processing the data locally. In such embodiments,external computing device 186 may receive and process the data todetermine whether an anomalous condition exists and, if so, whether tosend an alert notification to one or more mobile computing devices 184.1and 184.2 or take other actions.

In one aspect, external computing device 186 may additionally oralternatively be part of an insurer computing system (or facilitatecommunications with an insurer computer system), and as such may accessinsurer databases, execute algorithms, execute applications, accessremote servers, communicate with remote processors, etc., as needed toperform insurance-related functions. Such insurance-related functionsmay include assisting insurance customers in evaluating autonomousoperation features, limiting manual vehicle operation based upon risklevels, providing information regarding risk levels associated withautonomous and/or manual vehicle operation along routes, and/ordetermining repair/salvage information for damaged vehicles. Forexample, external computing device 186 may facilitate the receipt ofautonomous operation or other data from one or more mobile computingdevices 184.1-184.N, which may each be running a Data Application toobtain such data from autonomous operation features or sensors 120associated therewith.

In aspects in which external computing device 186 facilitatescommunications with an insurer computing system (or is part of such asystem), data received from one or more mobile computing devices184.1-184.N, and/or the smart home controllers 185 and/or personalelectronic devices 189, may include user credentials, which may beverified by external computing device 186 or one or more other externalcomputing devices, servers, etc. These user credentials may beassociated with an insurance profile, which may include, for example,insurance policy numbers, a description and/or listing of insuredassets, vehicle identification numbers of insured vehicles, addresses ofinsured structures, contact information, premium rates, discounts, etc.

In this way, data received from one or more mobile computing devices184.1-184.N, and/or the smart home controllers 185 and/or personalelectronic devices 189, may allow external computing device 186 touniquely identify each insured customer and/or whether each identifiedinsurance customer has installed the Data Application. In addition,external computing device 186 may facilitate the communication of theupdated insurance policies, premiums, rates, discounts, etc., toinsurance customers for their review, modification, and/or approval—suchas via wireless communication or data transmission to one or more mobilecomputing devices 184.1-184.N over one or more radio frequency links orwireless communication channels.

In some aspects, external computing device 186 may facilitate indirectcommunications between one or more of mobile computing devices 184,vehicles 182, smart home controllers 185, personal electronic devices189, and/or smart infrastructure component 188 via network 130 oranother suitable communication network, wireless communication channel,and/or wireless link. Smart infrastructure components 188 may beimplemented as any suitable type of traffic infrastructure componentsconfigured to receive communications from and/or to send communicationsto other devices, such as mobile computing devices 184 and/or externalcomputing device 186. Thus, smart infrastructure components 188 mayinclude infrastructure components 126 having infrastructurecommunication devices 124. For example, smart infrastructure component188 may be implemented as a traffic light, a railroad crossing signal, aconstruction notification sign, a roadside display configured to displaymessages, a billboard display, a parking garage monitoring device, etc.

In some embodiments, the smart infrastructure component 188 may includeor be communicatively connected to one or more sensors (not shown) fordetecting information relating to the condition of the smartinfrastructure component 188, which sensors may be connected to or partof the infrastructure communication device 124 of the smartinfrastructure component 188. The sensors (not shown) may generate datarelating to weather conditions, traffic conditions, or operating statusof the smart infrastructure component 188. The smart infrastructurecomponent 188 may be configured to receive the sensor data generated anddetermine a condition of the smart infrastructure component 188, such asweather conditions, road integrity, construction, traffic, availableparking spaces, etc.

In some aspects, smart infrastructure component 188 may be configured tocommunicate with one or more other devices directly and/or indirectly.For example, smart infrastructure component 188 may be configured tocommunicate directly with mobile computing device 184.2 via radio link183 d and/or with mobile computing device 184.1 via links 183 b and 183f utilizing network 130. As another example, smart infrastructurecomponent 188 may communicate with external computing device 186 vialinks 183 e and 183 f utilizing network 130. To provide someillustrative examples of the operation of the smart infrastructurecomponent 188, if smart infrastructure component 188 is implemented as asmart traffic light, smart infrastructure component 188 may change atraffic light from green to red (or vice-versa) or adjust a timing cycleto favor traffic in one direction over another based upon data receivedfrom the vehicles 182. If smart infrastructure component 188 isimplemented as a traffic sign display, smart infrastructure component188 may display a warning message that an anomalous condition (e.g., anaccident) has been detected ahead and/or on a specific roadcorresponding to the geographic location data.

FIG. 2 illustrates a block diagram of an exemplary mobile device 110,smart home controller 185, or an exemplary on-board computer 114consistent with the system 100 and the system 180. The mobile device110, smart home controller 185, or the on-board computer 114 may includea display 202, a GPS unit 206, a communication unit 220, anaccelerometer 224, one or more additional sensors (not shown), auser-input device (not shown), and/or, like the server 140, a controller204. In some embodiments, the mobile device 110 and on-board computer114 may be integrated into a single device, or either may perform thefunctions of both. The on-board computer 114 (or mobile device 110)interfaces with the sensors 120 to receive information regarding thevehicle 108 and its environment, which information is used by theautonomous operation features to operate the vehicle 108. Similarly, thesmart home controller 185 interfaces with a plurality of sensors disposeon and/or proximate to a smart home 187, which information is used bythe autonomous operation features to operate the smart home 187.

Similar to the controller 155, the controller 204 may include a programmemory 208, one or more microcontrollers or microprocessors (MP) 210, aRAM 212, and an I/O circuit 216, all of which are interconnected via anaddress/data bus 214. The program memory 208 includes an operatingsystem 226, a data storage 228, a plurality of software applications230, and/or a plurality of software routines 240. The operating system226, for example, may include one of a plurality of general purpose ormobile platforms, such as the Android™, iOS®, or Windows® systems,developed by Google Inc., Apple Inc., and Microsoft Corporation,respectively. Alternatively, the operating system 226 may be a customoperating system designed for autonomous vehicle operation using theon-board computer 114. The data storage 228 may include data such asuser profiles and preferences, application data for the plurality ofapplications 230, routine data for the plurality of routines 240, andother data related to the autonomous operation features. In someembodiments, the controller 204 may also include, or otherwise becommunicatively connected to, other data storage mechanisms (e.g., oneor more hard disk drives, optical storage drives, solid state storagedevices, etc.) that reside within the vehicle 108 and/or the smart home187.

As discussed with reference to the controller 155, it should beappreciated that although FIG. 2 depicts only one microprocessor 210,the controller 204 may include multiple microprocessors 210. Similarly,the memory of the controller 204 may include multiple RAMs 212 andmultiple program memories 208. Although FIG. 2 depicts the I/O circuit216 as a single block, the I/O circuit 216 may include a number ofdifferent types of I/O circuits. The controller 204 may implement theRAMs 212 and the program memories 208 as semiconductor memories,magnetically readable memories, or optically readable memories, forexample.

The one or more processors 210 may be adapted and configured to executeany of one or more of the plurality of software applications 230 or anyone or more of the plurality of software routines 240 residing in theprogram memory 204, in addition to other software applications. One ofthe plurality of applications 230 may be an autonomous vehicle operationapplication 232 that may be implemented as a series of machine-readableinstructions for performing the various tasks associated withimplementing one or more of the autonomous operation features accordingto the autonomous operation method 300, described further below.Similarly, one of the plurality of applications 230 may be a smart homeoperation application (not depicted) for performing various tasksassociated with implementing one or more of the autonomous operationfeatures of the smart home 187. Another of the plurality of applications230 may be an autonomous communication application 234 that may beimplemented as a series of machine-readable instructions fortransmitting and receiving autonomous operation information to or fromexternal sources via the communication module 220. Still anotherapplication of the plurality of applications 230 may include anautonomous operation monitoring application 236 that may be implementedas a series of machine-readable instructions for sending informationregarding autonomous operation of the vehicle to the server 140 via thenetwork 130. The Data Application for collecting, generating,processing, analyzing, transmitting, receiving, and/or acting uponautonomous operation feature data may also be stored as one of theplurality of applications 230 in the program memory 208 of the mobilecomputing device 110, the on-board computer 114, and/or the smart homecontroller 185, which may be executed by the one or more processors 210thereof.

The plurality of software applications 230 may call various of theplurality of software routines 240 to perform functions relating toautonomous vehicle and/or smart home operation, monitoring, orcommunication. One of the plurality of software routines 240 may be aconfiguration routine 242 to receive settings from the vehicle operatorand/or smart home occupant to configure the operating parameters of anautonomous operation feature. Another of the plurality of softwareroutines 240 may be a sensor control routine 244 to transmitinstructions to a sensor 120 and receive data from the sensor 120. Stillanother of the plurality of software routines 240 may be an autonomouscontrol routine 246 that performs a type of autonomous control, such ascollision avoidance, lane centering, speed control, fire prevention, ortemperature control. In some embodiments, the autonomous vehicleoperation application 232 may cause a plurality of autonomous controlroutines 246 to determine control actions required for autonomousvehicle operation.

Similarly, one of the plurality of software routines 240 may be amonitoring and reporting routine 248 that transmits informationregarding autonomous vehicle and/or smart home operation to the server140 via the network 130. Yet another of the plurality of softwareroutines 240 may be an autonomous communication routine 250 forreceiving and transmitting information between the vehicle 108, thesmart home 187, and external sources to improve the effectiveness of theautonomous operation features. Any of the plurality of softwareapplications 230 may be designed to operate independently of thesoftware applications 230 or in conjunction with the softwareapplications 230.

When implementing the exemplary autonomous operation method 300, thecontroller 204 of the on-board computer 114 may implement the autonomousvehicle operation application 232 to communicate with the sensors 120 toreceive information regarding the vehicle 108 and its environment andprocess that information for autonomous operation of the vehicle 108. Insome embodiments including external source communication via thecommunication component 122 or the communication unit 220, thecontroller 204 may further implement the autonomous communicationapplication 234 to receive information for external sources, such asother autonomous vehicles, smart infrastructure (e.g., electronicallycommunicating roadways, traffic signals, or parking structures),personal electronic devices, or other sources of relevant information(e.g., weather, traffic, local amenities). Some external sources ofinformation may be connected to the controller 204 via the network 130,such as the server 140 or internet-connected third-party databases (notshown). Although the autonomous vehicle operation application 232 andthe autonomous communication application 234 are shown as two separateapplications, it should be understood that the functions of theautonomous operation features may be combined or separated into anynumber of the software applications 230 or the software routines 240.

When implementing the autonomous operation feature monitoring method400, the controller 204 may further implement the autonomous operationmonitoring application 236 to communicate with the server 140 to provideinformation regarding autonomous operation. This may include informationregarding settings or configurations of autonomous operation features,data from the sensors 120 and/or sensors associated with the smart home187 regarding the environment, data from the sensors 120 and/or sensorsassociated with the smart home 187 regarding the response of the vehicle108 and/or the smart home 187 to its environment, respectively,communications sent or received using the communication component 122 orthe communication unit 220, operating status of the autonomous vehicleoperation application 232 and the autonomous communication application234, and/or control commands sent from the on-board computer 114 and/orthe smart home controller 185 to the control components (not shown) tooperate the vehicle 108 and/or the smart home 187.

In some embodiments, control commands generated by the on-board computer114 and/or the smart home controller 185 but not implemented may also berecorded and/or transmitted for analysis of how the autonomous operationfeatures would have responded to conditions if the features had beencontrolling the relevant aspect or aspects of vehicle operation. Theinformation may be received and stored by the server 140 implementingthe autonomous operation information monitoring application 141, and theserver 140 may then determine the effectiveness of autonomous operationunder various conditions by implementing the feature evaluationapplication 142, which may include an assessment of autonomous operationfeatures compatibility. The effectiveness of autonomous operationfeatures and the extent of their use may be further used to determineone or more risk levels associated with operation of the autonomousvehicle by the server 140.

In addition to connections to the sensors 120 that are external to themobile device 110 or the on-board computer 114, the mobile device 110 orthe on-board computer 114 may include additional sensors 120, such asthe GPS unit 206 or the accelerometer 224, which may provide informationregarding the vehicle 108 for autonomous operation and other purposes.Such sensors 120 may further include one or more sensors of a sensorarray 225, which may include, for example, one or more cameras,accelerometers, gyroscopes, magnetometers, barometers, thermometers,proximity sensors, light sensors, Hall Effect sensors, etc. The one ormore sensors of the sensor array 225 may be positioned to determinetelematics data regarding the speed, force, heading, and/or directionassociated with movements of the vehicle 108. Furthermore, thecommunication unit 220 may communicate with other autonomous vehicles,infrastructure, or other external sources of information to transmit andreceive information relating to autonomous vehicle operation. Thecommunication unit 220 may communicate with the external sources via thenetwork 130 or via any suitable wireless communication protocol network,such as wireless telephony (e.g., GSM, CDMA, LTE, etc.), Wi-Fi (802.11standards), WiMAX, Bluetooth, infrared or radio frequency communication,etc. Furthermore, the communication unit 220 may provide input signalsto the controller 204 via the I/O circuit 216. The communication unit220 may also transmit sensor data, device status information, controlsignals, or other output from the controller 204 to one or more externalsensors within the vehicle 108, mobile devices 110, on-board computers114, or servers 140.

The mobile device 110, the smart home controller 185, and/or theon-board computer 114 may include a user-input device (not shown) forreceiving instructions or information from the vehicle operator and/orsmart home occupant, such as settings relating to an autonomousoperation feature. The user-input device (not shown) may include a“soft” keyboard that is displayed on the display 202, an externalhardware keyboard communicating via a wired or a wireless connection(e.g., a Bluetooth keyboard), an external mouse, a microphone, or anyother suitable user-input device. The user-input device (not shown) mayalso include a microphone capable of receiving user voice input.

Data Application

The mobile device 110, the smart home controller 185 and/or on-boardcomputer 114 may run a Data Application to collect, transmit, receive,and/or process autonomous operation feature data. Such autonomousoperation feature data may include data directly generated by autonomousoperation features, such as control commands used in operating thevehicle 108 and/or the smart home 187. Similarly, such autonomousoperation feature data may include shadow control commands generated bythe autonomous operation features but not actually used in operating thevehicle and/or the smart home, such as may be generated when theautonomous operation features are disabled. The autonomous operationfeature data may further include non-control data generated by theautonomous operation features, such as determinations regardingenvironmental conditions in the vehicle operating environment in whichthe vehicle 108 and/or the smart home 187 operates (e.g., trafficconditions, construction locations, pothole locations, worn lanemarkings, corners with obstructed views, weather conditions, crimeconditions, etc.). The autonomous operation feature data may yet furtherinclude sensor data generated by (or derived from sensor data generatedby) sensors 120 and/or sensor associated with the smart home 185utilized by the autonomous operation features. For example, data fromLIDAR and ultrasonic sensors may be used by vehicles for autonomousoperation. As another example, an accelerometer may be used by smarthomes to detect an earthquake and autonomously initiate an appropriateresponse. Such data captures a much more detailed and completerepresentation of the conditions in which the vehicle 108 and/or smarthome 187 operates than traditional operation metrics (e.g., milesdriven) or non-autonomous telematics data (e.g., acceleration, position,and time).

Autonomous operation feature data may be processed and used by the DataApplication to determine information regarding the vehicle 108 and/orthe smart home 187, its operation, or its operating environment. Theautonomous operation feature data may further be communicated by theData Application to a server 140 via network 130 for processing and/orstorage. In some embodiments, the autonomous operation feature data (orinformation derived therefrom) may be transmitted directly via radiolinks 183 or indirectly via network 130 from the vehicle 108 and/or thesmart home 187 to other vehicles and/or smart homes (or to mobiledevices 110). By communicating information associated with theautonomous operation feature data to other nearby vehicles and/or smarthomes, the other vehicles and/or smart homes or their operators may makeuse of such data for routing, control, or other purposes. This may beparticularly valuable in providing detailed information regarding anoperating environment (e.g., traffic, accidents, flooding, ice, etc.)collected by a Data Application of an autonomous vehicle 108 and/or thesmart home 187 to a driver of a non-autonomous vehicle and/or anoccupant of a non-smart home via a Data Application of a mobile device110 associated with the driver and/or occupant. For example, ice patchesmay be identified by an autonomous operation feature of a vehiclecontroller 181.1 of vehicle 182.1 and transmitted via the DataApplication operating in the mobile computing device 184.1 over thenetwork 130 to the mobile computing device 184.2, where a warningregarding the ice patches may be presented to the driver of vehicle182.2. As another example, locations of emergency vehicles or accidentsmay be determined and communicated between vehicles 182, such as betweenan autonomous vehicle 182.1 and a traditional (non-autonomous) vehicle182.2.

In further embodiments, a Data Application may serve as an interfacebetween the user and an autonomous vehicle 108 and/or the smart home187, via the user's mobile device 110, the vehicle's on-board computer114, and/or the smart home controller 185. The user may interact withthe Data Application to locate, retrieve, or park the vehicle 108 and/orcontrol or monitor the vehicle 108 and/or the smart home 187. Forexample, the Data Application may be used to select a destination androute the vehicle 108 to the destination, which may include controllingthe vehicle to travel to the destination in a fully autonomous mode. Insome embodiments, the Data Application may further determine and/orprovide information regarding the vehicle 108, such as the operatingstatus or condition of autonomous operation features, sensors, or othervehicle components (e.g., tire pressure). In yet further embodiments,the Data Application may be configured to assess risk levels associatedwith operation based upon location, autonomous operation feature use(including settings), operating conditions, or other factors. Such riskassessment may be further used in recommending autonomous feature uselevels, generating warnings to a vehicle operator and/or smart homeoccupant, or adjusting an insurance policy associated with the vehicle108 and/or the smart home 187.

Data Applications may be installed and running on a plurality of mobiledevices 110, on-board computers 114, and/or smart home controllers 185in order to facilitate data sharing and other functions as describedherein. Additionally, such Data Applications may provide data to, andreceive data from, one or more servers 140. For example, a DataApplication running on a user's mobile device 110 may communicatelocation data to a server 140 via the network 130. The server 140 maythen process the data to determine a route, risk level, recommendation,or other action. The server 140 may then communicate the determinedinformation to the mobile device 110, the on-board computer 114, and/orthe smart home controller 185, which may cause the vehicle 108 and/orthe smart home 187 to operate in accordance with the determinedinformation (e.g., travel along a determined optimal route, initiatemeasures to prevent weather/environmental damage, etc.). Thus, the DataApplication may facilitate data communication between the front-endcomponents 102 and the back-end components 104, allowing more efficientprocessing and data storage.

Exemplary Autonomous Operation Method

FIG. 3 illustrates a flow diagram of an exemplary autonomous operationmethod 300, which may be implemented by the autonomous vehicle datasystem 100. The method 300 may begin when the controller 204 receives astart signal (block 302). The start signal may be a command from thevehicle operator through the user-input device to enable or engage oneor more autonomous operation features of the vehicle 108 and/or acommand from a smart home occupant through the user-input device toenable or engage one or more autonomous operation features of the smarthome 187. In some embodiments, the vehicle operator and/or smart homeoccupant may further specify settings or configuration details for theautonomous operation features. For fully autonomous vehicles, thesettings may relate to one or more destinations, route preferences, fuelefficiency preferences, speed preferences, or other configurablesettings relating to the operation of the vehicle 108. In someembodiments, fully autonomous vehicles may include additional featuresor settings permitting them to operate without passengers or vehicleoperators within the vehicle. For example, a fully autonomous vehiclemay receive an instruction to find a parking space within the generalvicinity, which the vehicle may do without the vehicle operator. Thevehicle may then be returned to a selected location by a request fromthe vehicle operator via a mobile device 110 or otherwise. This featuremay further be adapted to return a fully autonomous vehicle if lost orstolen.

For some autonomous vehicles and/or smart homes, the settings mayinclude enabling or disabling particular autonomous operation features,specifying thresholds for autonomous operation, specifying warnings orother information to be presented to the vehicle operator and/oroccupant, specifying autonomous communication types to send or receive,specifying conditions under which to enable or disable autonomousoperation features, or specifying other constraints on featureoperation. For example, a vehicle operator may set the maximum speed foran adaptive cruise control feature with automatic lane centering. Insome embodiments, the settings may further include a specification ofwhether the vehicle 108 should be operating as a fully or partiallyautonomous vehicle.

In embodiments where only one autonomous operation feature is enabled,the start signal may consist of a request to perform a particular task(e.g., autonomous parking or engage weather-proofing) or to enable aparticular feature (e.g., autonomous braking for collision avoidance orengage autonomous lighting that follows an occupant). In otherembodiments, the start signal may be generated automatically by thecontroller 204 based upon predetermined settings (e.g., when the vehicle108 exceeds a certain speed or is operating in low-light conditions). Insome embodiments, the controller 204 may generate a start signal whencommunication from an external source is received (e.g., when thevehicle 108 is on a smart highway, near another autonomous vehicle, or anational weather service issues an alert). In some embodiments, thestart signal may be generated by or received by the Data Applicationrunning on a mobile device 110, on-board computer 114 within the vehicle108, and/or smart home controller within the smart home 187. The DataApplication may further set or record settings for one or moreautonomous operation features of the vehicle 108 and/or the smart home187.

After receiving the start signal at block 302, the controller 204receives sensor data from the sensors 120, and/or sensors associatedwith the smart home 187, during operation (block 304). In someembodiments, the controller 204 may also receive information fromexternal sources through the communication component 122 or thecommunication unit 220. The sensor data may be stored in the RAM 212 foruse by the autonomous vehicle operation application 232. In someembodiments, the sensor data may be recorded in the data storage 228 ortransmitted to the server 140 via the network 130. The Data Applicationmay receive the sensor data, or a portion thereof, and store or transmitthe received sensor data. In some embodiments, the Data Application mayprocess or determine summary information from the sensor data beforestoring or transmitting the summary information. The sensor data mayalternately either be received by the controller 204 as raw datameasurements from one of the sensors 120 and/or sensors associated withthe smart home 187 or may be preprocessed by the sensor 120 and/orsensors associated with the smart home 187 prior to being received bythe controller 204. For example, a tachometer reading may be received asraw data or may be preprocessed to indicate vehicle movement orposition. As another example, a sensor 120 comprising a radar or LIDARunit may include a processor to preprocess the measured signals and senddata representing detected objects in 3-dimensional space to thecontroller 204.

The autonomous vehicle operation application 232 or other applications230 or routines 240 may cause the controller 204 to process the receivedsensor data in accordance with the autonomous operation features (block306). The controller 204 may process the sensor data to determinewhether an autonomous control action is required or to determineadjustments to the controls of the vehicle 108 and/or the smart home 187(i.e., control commands). For example, the controller 204 may receivesensor data indicating a decreasing distance to a nearby object in thevehicle's path and process the received sensor data to determine whetherto begin braking (and, if so, how abruptly to slow the vehicle 108).

As another example, the controller 204 may process the sensor data todetermine whether the vehicle 108 is remaining with its intended path(e.g., within lanes on a roadway). If the vehicle 108 is beginning todrift or slide (e.g., as on ice or water), the controller 204 maydetermine appropriate adjustments to the controls of the vehicle tomaintain the desired bearing. If the vehicle 108 is moving within thedesired path, the controller 204 may nonetheless determine whetheradjustments are required to continue following the desired route (e.g.,following a winding road). Under some conditions, the controller 204 maydetermine to maintain the controls based upon the sensor data (e.g.,when holding a steady speed on a straight road).

As still another example, the controller 204 may process the sensor datato determine whether fire containment and/or extinguishing response isactually containing and/or extinguishing a fire located at the smarthome 187. If the fire continues to spread, the controller 204 maydetermine appropriate adjustments to the containment and/orextinguishing response. If the fire is properly contained, thecontroller 204 may continue to monitor the fire containment and/orextinguishing response to prevent collateral damage.

In some embodiments, the Data Application may record information relatedto the processed sensor data, including whether the autonomous operationfeatures have determined one or more control actions to control thevehicle and/or details regarding such control actions. The DataApplication may record such information even when no control actions aredetermined to be necessary or where such control actions are notimplemented. Such information may include information regarding thevehicle operating environment determined from the processed sensor data(e.g., construction, other vehicles, pedestrians, anomalousenvironmental conditions, etc.). The information collected by the DataApplication may further include an indication of whether and/or how thecontrol actions are implemented using control components of the vehicle108 and/or the smart home 187.

When the controller 204 determines an autonomous control action isrequired (block 308), the controller 204 may cause the controlcomponents of the vehicle 108 to adjust the operating controls of thevehicle 108 and/or the smart home 187 achieve desired operation (block310). For example, the controller 204 may send a signal to open or closethe throttle of the vehicle 108 to achieve a desired speed.Alternatively, the controller 204 may control the steering of thevehicle 108 to adjust the direction of movement. In some embodiments,the vehicle 108 may transmit a message or indication of a change invelocity or position using the communication component 122 or thecommunication module 220, which signal may be used by other autonomousvehicles to adjust their controls. As discussed elsewhere herein, thecontroller 204 may also log or transmit the autonomous control actionsto the server 140 via the network 130 for analysis. In some embodiments,an application (which may be a Data Application) executed by thecontroller 204 may communicate data to the server 140 via the network130 or may communicate such data to the mobile device 110 for furtherprocessing, storage, transmission to nearby vehicles, smart homes,infrastructure, and/or communication to the server 140 via network 130.

The controller 204 may continue to receive and process sensor data atblocks 304 and 306 until an end signal is received by the controller 204(block 312). The end signal may be automatically generated by thecontroller 204 upon the occurrence of certain criteria (e.g., thedestination is reached, environmental conditions require manualoperation of the vehicle 108 by the vehicle operator, or anoccupant-defined end condition is satisfied). Alternatively, the vehicleoperator and/or smart home occupant may pause, terminate, or disable theautonomous operation feature or features using the user-input device orby manually operating the vehicle's controls, such as by depressing apedal or turning a steering instrument, and/or by manually interactingwith smart equipment disposed in the smart home 187. When the autonomousoperation features are disabled or terminated, the controller 204 mayeither continue operation without the autonomous features or may shutoff the vehicle 108 and/or various smart equipment, depending upon thecircumstances.

Where control of the vehicle 108 must be returned to the vehicleoperator and/or control of the smart home 187 must be returned to thesmart home occupant, the controller 204 may alert the vehicle operatorand/or smart home occupant in advance of returning to manual operation.The alert may include a visual, audio, or other indication to obtain theattention of the vehicle operator and/or smart home occupant. In someembodiments, the controller 204 may further determine whether thevehicle operator is capable of resuming manual operation beforeterminating autonomous operation. If the vehicle operator is determinednot to be capable of resuming operation, the controller 204 may causethe vehicle to stop or take other appropriate action.

The autonomous operation features may generate and implement controldecisions relating to the control of the motive, steering, and stoppingcomponents of the vehicle 108 and/or the various components of the smarthome 187. The control decisions may include or be related to controlcommands issued by the autonomous operation features to control suchcontrol components of the vehicle 108 and/or the smart home 187 duringoperation. In some embodiments, control decisions may include decisionsdetermined by the autonomous operation features regarding controlcommands such feature would have issued under the conditions thenoccurring, but which control commands were not issued or implemented.For example, an autonomous operation feature may generate and recordshadow control decisions it would have implemented if engaged to operatethe vehicle 108 and/or the smart home 187 even when the feature isdisengaged (or engaged using other settings from those that wouldproduce the shadow control decisions).

Data regarding the control decisions actually implemented and/or theshadow control decisions not implemented to control the vehicle 108and/or the smart home 187 may be recorded for use in assessingautonomous operation feature effectiveness, accident and/or eventreconstruction and fault determination, feature use or settingsrecommendations, risk determination and insurance policy adjustments, orother purposes as described elsewhere herein. For example, actualcontrol decisions may be compared against control decisions that wouldhave been made by other systems, software versions, or with additionalsensor data or communication data.

As used herein, the terms “preferred” or “preferably made” controldecisions mean control decisions that optimize some metric associatedwith risk under relevant conditions. Such metric may include, amongother things, a statistical correlation with one or more risks (e.g.,risks related to a vehicle collision) or an expected value associatedwith risks (e.g., a risk-weighted expected loss associated withpotential vehicle accidents). The preferably made, or preferred orrecommended, control decisions discussed herein may include controldecisions or control decision outcomes that are less risky, have lowerrisk or the lowest risk of all the possible or potential controldecisions given various operating conditions, and/or are otherwiseideal, recommended, or preferred based upon various operatingconditions, including autonomous system or feature capability; currentroad, environmental or weather, traffic, or construction conditionsthrough which the vehicle is traveling; and/or current versions ofautonomous system software or components that the autonomous vehicle isequipped with and using.

The preferred or recommended control decisions may result in the lowestlevel of potential or actual risk of all the potential or possiblecontrol decisions given a set of various operating conditions and/orsystem features or capabilities. Alternatively, the preferred orrecommended control decisions may result in a lower level of potentialor actual risk (for a given set of operating conditions) to theautonomous vehicle and passengers, and other people or vehicles, thansome of the other potential or possible control decisions that couldhave been made by the autonomous system or feature.

Exemplary Monitoring Method

FIG. 4A is a flow diagram depicting an exemplary autonomous operationmonitoring method 400, which may be implemented by the autonomousvehicle data system 100. The method 400 monitors the operation of thevehicle 108, the smart home 187, and/or the personal electronic device189 and transmits information regarding the vehicle 108, the smart home187, and/or the personal electronic device 189 to the server 140, whichinformation may then be used to determine autonomous operation featureusage or effectiveness. The method 400 may be used for monitoring thestate of the vehicle 108, the smart home 187, and/or the personalelectronic device 189, for providing data to other vehicles 182 and/orsmart homes, for responding to emergencies or unusual situations duringoperation, for testing autonomous operation features in a controlledenvironment, for determining actual feature use during operation outsidea test environment, for assessment of feature operation, and/or forother purposes described herein. In alternative embodiments, the method400 may be implemented whenever the vehicle 108, the smart home 187,and/or the personal electronic device 189 is in operation (manual orautonomous) or only when the autonomous operation features are enabled.

The method 400 may likewise be implemented as either a real-timeprocess, in which information regarding the vehicle 108, the smart home187, and/or the personal electronic device 189 is communicated to theserver 140 while monitoring is ongoing, or as a periodic process, inwhich the information is stored within the vehicle 108, the smart home187, and/or the personal electronic device 189 and communicated to theserver 140 at intervals (e.g., upon completion of a trip or when anincident occurs, when a loss-event occurs, etc.). In some embodiments,the method 400 may communicate with the server 140 in real-time whencertain conditions exist (e.g., when a sufficient data connectionthrough the network 130 exists or when no roaming charges would beincurred). In further embodiments, a Data Application executed by themobile device 110, the on-board computer 114, the smart home controller185, and/or the personal electronic device 189 may perform suchmonitoring, recording, and/or communication functions, including any ofthe functions described below with respect to blocks 402-434.

The method 400 may begin when the controller 204 receives an indicationof vehicle, smart home, and/or personal electronic device operation(block 402). The indication may be generated when the vehicle 108 isstarted, when the smart home 187 becomes occupied, or when an autonomousoperation feature is enabled by the controller 204 or by input from thevehicle operator and/or smart home occupant, as discussed above. Inresponse to receiving the indication, the controller 204 may create atimestamp (block 404). The timestamp may include information regardingthe date, time, location, operating environment, condition, andautonomous operation feature settings or configuration information. Thedate and time may be used to identify one vehicle trip or one period ofautonomous operation feature use, in addition to indicating risk levelsdue to various factors, such as traffic, weather, and so on.

The additional location and environmental data may include informationregarding the position of the vehicle 108 from the GPS unit 206 and itssurrounding environment (e.g., road conditions, weather conditions,nearby traffic conditions, type of road, construction conditions,presence of pedestrians, presence of other obstacles, availability ofautonomous communications from external sources, etc.). Conditioninformation may include information regarding the type, make, and modelof the vehicle 108, the age or mileage of the vehicle 108, the status ofvehicle equipment (e.g., tire pressure, non-functioning lights, fluidlevels, etc.), the type, make, and model of smart devices within thesmart home 187, or other information relating to the vehicle 108, thesmart home 187, and/or the personal electronic device 189. In someembodiments, condition information may further include informationregarding the sensors 120, such as type, configuration, or operationalstatus (which may be determined, for example, from analysis of actual ortest data from the sensors). In some embodiments, the timestamp may berecorded on the client device 114, the mobile device 110, or the server140.

The autonomous operation feature settings may correspond to informationregarding the autonomous operation features, such as those describedabove with reference to the autonomous operation method 300. Theautonomous operation feature configuration information may correspond toinformation regarding the number and type of the sensors 120 (which mayinclude indications of manufacturers and models of the sensors 120), thedisposition of the sensors 120 within the vehicle 108, the smart home187, and/or the personal electronic device 189 (which may includedisposition of sensors 120 within one or more mobile devices 110), theone or more autonomous operation features (e.g., the autonomous vehicleoperation application 232 or the software routines 240), autonomousoperation feature control software, versions of the softwareapplications 230 or routines 240 implementing the autonomous operationfeatures, or other related information regarding the autonomousoperation features.

For example, the configuration information may include the make andmodel of the vehicle 108 (indicating installed sensors 120 and the typeof on-board computer 114), an indication of smart devices and the typeof smart home controller 185 within the smart home 187, an indication ofa malfunctioning or obscured sensor 120 in part of the vehicle 108, thesmart home 187, and/or the personal electronic device 189, informationregarding additional after-market sensors 120 installed within thevehicle 108, the smart home 187, and/or the personal electronic device189, a software program type and version for a control program installedas an application 230 on the on-board computer 114, the smart homecontroller 185, and/or the personal electronic device 189, and softwareprogram types and versions for each of a plurality of autonomousoperation features installed as applications 230 or routines 240 in theprogram memory 208 of the on-board computer 114, the smart homecontroller 185, and/or the personal electronic device 189.

During operation, the sensors 120 may generate sensor data regarding thevehicle 108, the smart home 187, and/or the personal electronic device189 and its environment, which may include other vehicles 182 or smarthomes within the operating environment of the vehicle 108, the smarthome 187, and/or the personal electronic device 189. In someembodiments, one or more of the sensors 120 may preprocess themeasurements and communicate the resulting processed data to theon-board computer 114, the smart home controller 185, and/or thepersonal electronic device 189 and/or the mobile device 110. Thecontroller 204 may receive sensor data from the sensors 120 (block 406).The sensor data may include information regarding the vehicle'sposition, speed, acceleration, direction, and responsiveness tocontrols.

The sensor data may further include information regarding the locationand movement of obstacles or obstructions (e.g., other vehicles,buildings, barriers, pedestrians, animals, trees, or gates), weatherconditions (e.g., precipitation, wind, visibility, or temperature), roadconditions (e.g., lane markings, potholes, road material, traction, orslope), signs or signals (e.g., traffic signals, construction signs,building signs or numbers, or control gates), or other informationrelating to the operating environment. In some embodiments, sensors 120may indicate the number of passengers within the vehicle 108 and/oroccupants within the smart home 187, including an indication of whetherthe vehicle and/or smart home is entirely empty.

In addition to receiving sensor data from the sensors 120, in someembodiments the controller 204 may receive autonomous communication datafrom the communication component 122 or the communication module 220(block 408). The communication data may include information from otherautonomous vehicles (e.g., sudden changes to vehicle speed or direction,intended vehicle paths, hard braking, vehicle failures, collisions, ormaneuvering or stopping capabilities), infrastructure (road or laneboundaries, bridges, traffic signals, control gates, or emergencystopping areas), other smart homes (e.g., the presence of potentiallyhazardous conditions), or other external sources (e.g., map databases,weather databases, or traffic and accident databases). In someembodiments, the communication data may include data from non-autonomousvehicles and/or homes, which may include data regarding operation oranomalies within the operating environment determined by a DataApplication operating on a mobile device 110, on-board computer 114, thesmart home controller 185, and/or the personal electronic device 189.The communication data may be combined with the received sensor datareceived to obtain a more robust understanding of the operatingenvironment.

For example, the server 140 or the controller 204 may combine sensordata indicating frequent changes in speed relative to tachometric datawith map data relating to a road upon which the vehicle 108 is travelingto determine that the vehicle 108 is in an area of hilly terrain. Asanother example, weather data indicating recent snowfall in the vicinityof the vehicle 108 may be combined with sensor data indicating frequentslipping or low traction to determine that the vehicle 108 is travelingon a snow-covered or icy road.

The controller 204 may process the sensor data, the communication data,and the settings or configuration information to determine whether anincident has occurred (block 410). As used herein, an “incident” is anoccurrence during operation of an autonomous vehicle, smart home, and/orpersonal electronic device outside of normal safe operating conditions,such that one or more of the following occurs: (i) there is aninterruption of ordinary operation, (ii) there is damage to the vehicle,smart home, personal electronic device or other property, (iii) there isinjury to a person, (iv) the conditions require action to be taken by anoperator, occupant, autonomous operation feature, pedestrian, or otherparty to avoid damage or injury, and/or (v) an anomalous condition isdetected that requires an adjustment outside of ordinary vehicleoperation. Incidents from categories (ii) and (iii) above may beconsidered “loss-events.” Incidents may include collisions, hardbraking, hard acceleration, evasive maneuvering, loss of traction,detection of objects within a threshold distance from the vehicle 108,alerts presented to the vehicle operator, component failure,inconsistent readings from sensors 120, or attempted unauthorized accessto the on-board computer by external sources. Incidents may also includeaccidents, vehicle breakdowns, flat tires, empty fuel tanks, or medicalemergencies. Incidents may further include identification ofconstruction requiring the vehicle to detour or stop, hazardousconditions (e.g., fog or road ice), or other anomalous environmentalconditions.

In some embodiments, the controller 204 may anticipate or project anexpected incident based upon sensor or external data, allowing thecontroller 204 to send control signals to minimize the negative effectsof the incident. For example, the controller 204 may cause the vehicle108 to slow and move to the shoulder of a road immediately beforerunning out of fuel. As another example, adjustable seats within thevehicle 108 may be adjusted to better position vehicle occupants inanticipation of a collision, windows may be opened or closed, or airbagsmay be deployed. As yet another example, storm shutters for windows ofthe smart home 187 may be activated in anticipation of a weather event.

When an incident is determined to have occurred (block 412), informationregarding the incident and the vehicle, smart home, and/or personalelectronic device status may be recorded (block 414), either in the datastorage 228 or the database 146. The information recorded may includesensor data, communication data, and settings or configurationinformation prior to, during, and immediately following the incident. Insome embodiments, a preliminary determination of fault may also beproduced and stored. The information may further include a determinationof whether the vehicle 108, the smart home 187, and/or the personalelectronic device 189 has continued operating (either autonomously ormanually) or whether the vehicle 108, the smart home 187, and/or thepersonal electronic device 189 is capable of continuing to operate incompliance with applicable safety and legal requirements. If thecontroller 204 determines that the vehicle 108, the smart home 187,and/or the personal electronic device 189 has discontinued operation oris unable to continue operation (block 416), the method 400 mayterminate. If the vehicle 108, the smart home 187, and/or the personalelectronic device 189 continues operation, then the method 400 maycontinue as described below with reference to block 418.

FIG. 4B illustrates an alternative portion of the method 400 followingan incident. When an incident is determined to have occurred (block412), the controller 204 or the server 140 may record status andoperating information (block 414), as above. In some instances, theincident may interrupt communication between the vehicle 108, the smarthome 187, and/or the personal electronic device 189 and the server 140via network 130, such that not all information typically recorded willbe available for recordation and analysis by the server 140. Based uponthe recorded data, the server 140 or the controller 204 may determinewhether assistance may be needed at the location of the vehicle 108, thesmart home 187, and/or the personal electronic device 189 (block 430).For example, the controller 204 may determine that a head-on collisionhas occurred based upon sensor data (e.g., airbag deployment, automaticmotor shut-off, LIDAR data indicating a collision, etc.) and may furtherdetermine based upon information regarding the speed of the vehicle 108and other information that medical, police, and/or towing services willbe necessary. The determination that assistance is needed may furtherinclude a determination of types of assistance needed (e.g., police,ambulance, fire, towing, vehicle maintenance, fuel delivery, etc.). Thisdetermination may include analysis of the type of incident, the sensordata regarding the incident (e.g., images from outward facing or inwardfacing cameras installed within the vehicle, the smart home, and/or thepersonal electronic device, identification of whether any passengersand/or occupants were present within the vehicle and/or smart home,determination of whether any pedestrians or passengers in other vehicleswere involved in the incident, etc.). The determination of whetherassistance is needed may further include information regarding thedetermined status of the vehicle 108, the smart home 187, and/or thepersonal electronic device 189.

In some embodiments, the determination regarding whether assistance isneeded may be supplemented by a verification attempt, such as a phonecall or communication through the on-board computer 114, the smart homecontroller 189, and/or the personal electronic device 189. Where theverification attempt indicates assistance is required or communicationattempts fail, the server 140 or controller 204 would then determinethat assistance is needed, as described above. For example, whenassistance is determined to be needed following an accident involvingthe vehicle 108 and/or a loss-event at the smart home 187, the server140 may direct an automatic telephone call to a mobile telephone numberassociated with the vehicle 108, the smart home 187, and/or the personalelectronic device 189 and/or operator and/or occupant thereof. If noresponse is received, or if the respondent indicates assistance isrequired, the server 140 may proceed to cause a request for assistanceto be generated.

When assistance is determined to be needed (block 432), the controller204 or the server 140 may send a request for assistance (block 434). Therequest may include information regarding the vehicle 108, the smarthome 187, and/or the personal electronic device 189, such as thelocation, the type of assistance required, other vehicles or homesinvolved in the incident, pedestrians involved in the incident, vehicleoperators or passengers involved in the incident, and/or other relevantinformation. The request for assistance may include telephonic, data, orother requests to one or more emergency or vehicular service providers(e.g., local police, fire departments, state highway patrols, emergencymedical services, public or private ambulance services, hospitals,towing companies, roadside assistance services, vehicle rental services,local claims representative offices, etc.). After sending a request forassistance (block 434) or when assistance is determined not to be needed(block 432), the controller 204 or the server 140 may next determinewhether the vehicle 108, the smart home 187, and/or the personalelectronic device 189 is operational (block 416), as described above.The method 400 may then end or continue as indicated in FIG. 4A.

In some embodiments, the controller 204 may further determineinformation regarding the likely cause of a collision or other incident.Alternatively, or additionally, the server 140 may receive informationregarding an incident from the on-board computer 114, the smart homecontroller 185, and/or the personal electronic device 189 and determinerelevant additional information regarding the incident from the sensordata. For example, the sensor data may be used to determine the pointsof impact on the vehicle 108 and another vehicle involved in acollision, the relative velocities of each vehicle, the road conditionsat the time of the incident, and the likely cause or the party likely atfault. This information may be used to determine risk levels associatedwith autonomous vehicle operation, as described below, even where theincident is not reported to the insurer.

The controller 204 may determine whether a change or adjustment to oneor more of the settings or configuration of the autonomous operationfeatures has occurred (block 418). Changes to the settings may includeenabling or disabling an autonomous operation feature or adjusting thefeature's parameters (e.g., resetting the speed on an adaptive cruisecontrol feature). For example, an operator may selectively enable ordisable autonomous operation features such as automatic braking, lanecentering, temperature control, or even fully autonomous operation atdifferent times. If the settings or configuration are determined to havechanged, the new settings or configuration may be recorded (block 422),either in the data storage 228 or the database 146. For example, theData Application may log autonomous operation feature use and changes ina log file, including timestamps associated with the features in use.

Next, the controller 204 may record the operating data relating to thevehicle 108, the smart home 187, and/or the personal electronic device189 in the data storage 228 or communicate the operating data to theserver 140 via the network 130 for recordation in the database 146(block 424). The operating data may include the settings orconfiguration information, the sensor data, and/or the communicationdata discussed above. In some embodiments, operating data related tonormal autonomous operation of the vehicle 108, the smart home 187,and/or the personal electronic device 189 may be recorded. In otherembodiments, only operating data related to incidents of interest may berecorded, and operating data related to normal operation may not berecorded. In still other embodiments, operating data may be stored inthe data storage 228 until a sufficient connection to the network 130 isestablished, but some or all types of incident information may betransmitted to the server 140 using any available connection via thenetwork 130.

The controller 204 may then determine whether operation of the vehicle108, the smart home 187, and/or the personal electronic device 189remains ongoing (block 426). In some embodiments, the method 400 mayterminate when all autonomous operation features are disabled, in whichcase the controller 204 may determine whether any autonomous operationfeatures remain enabled. When the vehicle 108 is determined to beoperating (or operating with at least one autonomous operation featureenabled), the method 400 may continue through blocks 406-426 untiloperation has ended. When the vehicle 108, the smart home 187, and/orthe personal electronic device 189 is determined to have ceasedoperating (or is operating without autonomous operation featuresenabled), the controller 204 may record the completion of operation(block 428), either in the data storage 228 or the database 146. In someembodiments, a second timestamp corresponding to the completion ofoperation may likewise be recorded, as above.

Exemplary Incident Response Methods

FIG. 5 illustrates a flow diagram of an exemplary incident responsemethod 500 for detecting and responding to incidents involving a vehicle108, a smart home 187, and/or a personal electronic device 189 whileengaged in fully autonomous operation or in the absence of an operatorand/or an occupant. The vehicle 108, the smart home 187, and/or thepersonal electronic device 189 may be operating in a fully autonomousmode of operation without any control decisions being made by a vehicleoperator and/or smart home occupant, excluding navigation decisions,such as selection of a destination or route, and/or preferencedecisions, such as selection of a desired temperature for a room. Insome embodiments, the vehicle 108 and/or the smart home 187 may beoperating without any passengers and/or occupants or with onlypassengers and/or occupants who are physically or legally unable tooperate the vehicle 108 and/or the smart home 187 in a manual orsemi-autonomous mode of operation (e.g., children, persons sufferingacute illness, physically handicapped persons, intoxicated or otherwiseimpaired persons, etc.). Alternatively, the vehicle 108 may be parked ina non-operating state. Particularly when the vehicle 108 and/or thesmart home 187 is unoccupied, detection and response to collisions orother incidents interrupting ordinary autonomous operation poseparticular challenges that do not arise during manual operation orsemi-autonomous operation. Such incidents are typically unintentionaland unexpected, and such incidents frequently coincide with damage tosensor components (including sensors 120) or structural components thatare essential for safe operation of the vehicle and/or smart home.Additionally, autonomous vehicles and/or smart homes may be incapable ofperforming actions typically performed by operators and/or occupants inresponse to manual incidents (e.g., moving a vehicle out of a trafficlane, replacing a tire, moving debris, assessing damage, providinginformation to authorities, etc.). The incident response method 500addresses at least these issues.

The incident response method 500 may begin by monitoring the conditionof the vehicle 108, the smart home 187, and/or the personal electronicdevice 189 (block 502), which may include monitoring operating data fromthe vehicle 108, the smart home 187, and/or the personal electronicdevice 189 during autonomous operation. If an indication of an unusualcondition is detected (block 504), further analysis may be performed todetermine whether an incident has occurred (block 506). If an incident(or an incident having sufficient impact upon operation of the vehicle108, the smart home 187, and/or the personal electronic device 189) isdetermined to have occurred (block 508), damage to the vehicle 108, thesmart home 187, the personal electronic device 189, and/or to otherobjects may be assessed (block 510). Such assessment may includedetermining the operating capabilities of the vehicle 108, the smarthome 187, and/or the personal electronic device 189, which may bediminished by damage to sensors 120 or other components.

Based upon the determination of damage, one or more responses to theincident may then be determined (block 512). The vehicle 108, the smarthome 187, and/or the personal electronic device 189 then implements theone or more responses to address the incident (block 514). In someembodiments, additional responses may be implemented by a server 140 orother device. The method 500 then determines whether monitoring shouldcontinue (block 516). If the vehicle 108, the smart home 187, and/or thepersonal electronic device 189 is continuing to operate or it isotherwise determined that monitoring should continue, the method 500continues to monitor the autonomous operation (block 502). If operationof the vehicle 108, the smart home 187, and/or the personal electronicdevice 189 has concluded, the method 500 terminates.

If no indication of an unusual condition is detected (block 504) or noincident having a sufficient impact upon autonomous operation isdetermined to have occurred (block 508), the method 500 may continue tomonitor operation of the vehicle 108, the smart home 187, and/or thepersonal electronic device 189 as long as vehicle operation continues orit is otherwise determined that monitoring should continue (block 516).Although the method 500 is described with reference to the on-boardcomputer 114, the smart home controller 185, and/or the personalelectronic device 189 for simplicity, the described method may bereadily modified for implementation by other systems or devices,including one or more of mobile devices 110 and/or servers 140.

At block 502, the on-board computer 114, the smart home controller 185,and/or the personal electronic device 189 of the vehicle 108 and/or thesmart home 187 may monitor the condition of the vehicle 108, the smarthome 187, and/or the personal electronic device 189. This may includereceiving and processing operating data from one or more sensors 120and/or other components within the vehicle 108, the smart home 187,and/or the personal electronic device 189. The on-board computer 114,the smart home controller 185, and/or the personal electronic device 189may begin monitoring autonomous operation automatically whenever thevehicle 108, the smart home 187, and/or the personal electronic device189 is started and/or becomes occupied, or whenever the vehicle 108, thesmart home 187, and/or the personal electronic device 189 begins fullyautonomous operation. Alternatively, the on-board computer 114, thesmart home controller 185, and/or the personal electronic device 189 maybegin monitoring when it detects that the vehicle 108, the smart home187, and/or the personal electronic device 189 is being operated in afully autonomous mode without passengers and/or occupants. In someembodiments, the on-board computer 114, the smart home 187, and/or thepersonal electronic device 189 may monitor the condition of the vehicle108, the smart home 187, and/or the personal electronic device 189 whennot in use, such as while the vehicle 108 is parked and/or while thesmart home 187 is unoccupied. In such embodiments, the on-board computer114, the smart home 187, and/or the personal electronic device 189, maybegin and/or continue monitoring the vehicle, smart home, and/orpersonal electronic device condition when the vehicle 108 is parked orshut down, or monitoring may begin and/or continue when the operatorexits the vehicle 108 and/or egresses the smart home 187.

To conserve energy and/or processor usage (particularly in electricvehicles or when parked, and/or smart homes operating on power derivedfrom a local generator), the on-board computer 114, the smart homecontroller 185, and/or the personal electronic device 189 may monitorthe vehicle 108, the smart home 187, and/or the personal electronicdevice 189 with a limited set of operating data or data from a limitedset of sensors 120. In further embodiments, the on-board computer 114,the smart home controller 185, and/or the personal electronic device 189may monitor the vehicle, smart home, and/or personal electronic devicecondition by comparing received operating data at time intervals longerthan the time intervals between ordinary sensor data readings used forautonomous operation. For example, the on-board computer 114, the smarthome controller 185, and/or the personal electronic device 189 mayprocess received operating data every thirty seconds or every minute todetect indications of incidents that may affect autonomous operation. Insome embodiments, the on-board computer 114, the smart home 187, and/orthe personal electronic device 189 may control one or more of thesensors 120 to generate sensor data at such intervals, particularlywhere the vehicle 108 is not presently operating (e.g., when parked)and/or when the smart home 187 is unoccupied.

The on-board computer 114, the smart home controller 185, and/or thepersonal electronic device 189 may monitor the operating data forindications of unusual conditions that are indicative of a likelihood ofan incident, which may require further assessment and response. Suchindications of unusual conditions may include discontinuities in theoperating data, divergence between sensor data from one or more sensorsand control data from one or more autonomous operation features, aplurality of sensor malfunctions, sudden sensor failure (particularlywhen multiple sensors fail at substantially the same time), and/orsensed conditions associated with incidents (e.g., distance to a sensedobject reaching zero, unexpected lane departures, etc.). In someembodiments, indications of unusual conditions may be directly receivedfrom sensors 120, autonomous operation features (i.e., hardware orsoftware components performing particular aspects of autonomousoperation), and/or other components of the vehicle 108, the smart home187, and/or the personal electronic device 189 as error signals oralerts. For example, a sensor 120 may perform a self-diagnostic routineat start-up or periodically and may further send an alert to theon-board computer 114, the smart home controller 185, and/or thepersonal electronic device 189 when the sensor 120 is self-determined tobe damaged or otherwise malfunctioning.

Indications of unusual conditions may include data points that areassociated with a likelihood of a collision or other incident requiringa response, even though such indications may be insufficient todetermine such response without further analysis. For example,unexpected or inaccurate sensor data from a nonessential sensor 120 maybe caused by damage or by temporary obstruction (e.g., by ice or dirtaccumulation).

In yet further embodiments, an indication of an unusual condition may bedetermined or received with respect to another vehicle, smart home,pedestrian, or object within the current environment of the vehicle 108,the smart home 187, and/or the personal electronic device 189. Suchindication may include information regarding an occurrence or likelihoodof an incident not directly involving the vehicle 108, the smart home187, and/or the personal electronic device 189. For example, theincident may involve another autonomous, semi-autonomous, or traditionalvehicle within a predetermined or variable distance of the vehicle 108.In a particular embodiment, the vehicle 108, the smart home 187, and/orthe personal electronic device 189 may receive an autonomouscommunication message including the indication of the unusual conditionfrom another vehicle and/or smart home via short-range wirelesscommunication transmission and via the communication component 122. Forexample, the other vehicle and/or smart home may automatically send adistress signal upon determining it has been involved in a collision orotherwise detects an emergency condition, which distress signal may bereceived and processed by the vehicle 108, the smart home 187, and/orthe personal electronic device 189 to determine and implement anappropriate response.

At block 504, the on-board computer 114, the smart home controller 185,and/or the personal electronic device 189 may determine whether anyindications of unusual conditions have been detected. If no suchindication of unusual conditions have been identified by the on-boardcomputer 114, the smart home controller 185, and/or the personalelectronic device 189, the method 500 may continue by determiningwhether to continue monitoring (block 516), as discussed below. If oneor more unusual conditions are identified by the on-board computer 114,the smart home controller 185, and/or the personal electronic device189, the method 500 may instead continue to determine whether anincident requiring a response has occurred (block 506).

At block 506, the on-board computer 114, the smart home controller 185,and/or the personal electronic device 189 determines whether an incidenthas occurred. This determination may include determining the type ofincident, as well as determining whether the incident requires aresponse. The on-board computer 114, the smart home controller 185,and/or the personal electronic device 189 may receive or collectadditional data regarding the vehicle 108, the smart home 187, and/orthe personal electronic device 189 or the autonomous environment for thedetermination. In some embodiments, this may include obtaining oraccessing additional operating data that had been previously generated.In further embodiments, the on-board computer 114, the smart homecontroller 185, and/or the personal electronic device 189 may cause oneor more sensors 120 or other components to generate additional operatingdata for the determination. In yet further embodiments, the on-boardcomputer 114, the smart home controller 185, and/or the personalelectronic device 189 may obtain communication data from other vehicles,smart homes, infrastructure components, or other data sources. Theoperating data to be used in determining whether an incident hasoccurred may be selected in part based upon the one or more identifiedindications of unusual conditions. For example, the on-board computer114, the smart home controller 185, and/or the personal electronicdevice 189 may activate additional sensors 120 positioned in the samearea of a vehicle and/or smart home as an identified potentiallymalfunctioning sensor 120 in order to obtain additional relevant dataregarding the identified sensor 120.

The on-board computer 114, the smart home controller 185, and/or thepersonal electronic device 189 may determine whether an incident hasoccurred by analyzing the operating data and/or other data to identifyincidents, such as collisions with other vehicles, infrastructure,pedestrians, animals, or other objects. Other incidents that may bedetermined may include component failure events (e.g., tire blowouts,sensor failure, etc.), software malfunctions (e.g., hacking attempts,cyber-attacks, corrupted software, unresponsive routines, etc.),impassible roadways (e.g., due to accidents, police action, flooding,debris, etc.), severe weather (e.g., dense fog, high winds, or otherconditions preventing autonomous operation), and/or other incidents thatmay require a response outside of the ordinary operation of the vehicle108, the smart home 187, and/or the personal electronic device 189.Although incidents are described as directly affecting the operation ofthe vehicle 108, the smart home 187, and/or the personal electronicdevice 189, some embodiments may include determining whether incidentsare occurring that impact the operation of another vehicle, smart home,personal electronic device, or person in the operating environment ofthe vehicle 108, the smart home 187, and/or the personal electronicdevice 189.

Determining the occurrence of an incident may include determining a typeof incident (e.g., collision, component failure, software malfunction,impassible roadway, severe weather, etc.). In some embodiments,determining the occurrence of an incident may further includedetermining whether the incident has a sufficient impact on autonomousoperation to require a response (i.e., is the incident of sufficientseverity or urgency as to require an immediate response). In yet furtherembodiments, determining an incident requires a response may includedetermining that the incident increases one or more risks associatedwith autonomous operation above a maximum threshold level for safeoperation of the vehicle 108, the smart home 187, and/or the personalelectronic device 189 in a fully autonomous mode. Determining anincident requires a response may further include determining whether alegal or other obligation requires a response, such as reporting acollision or remaining at the location of a collision. In someinstances, determining an incident requires a response may includedetermining whether a response is required to assist an injuredpedestrian, passenger of another vehicle, an occupant, and/or thatsimilar assistance is needed.

The on-board computer 114, the smart home controller 185, and/or thepersonal electronic device 189 may determine an occurrence (which mayinclude the severity or impact) of the incident based upon an analysisof the obtained data. This determination may include comparing theobtained operating data and/or other data with expected or baseline datato determine whether the obtained data is outside an ordinary orexpected range. This determination may further include comparing datafrom a plurality of sources to determine inconsistencies and identifysources of such inconsistencies (e.g., identifying which of a pluralityof sensors 120 is malfunctioning when inconsistent data is received). Insome embodiments, this determination may further include analyzing orreconstructing a portion of a time period associated with the unusualcondition to determine whether the incident has occurred. For example,the on-board computer 114 and/or smart home controller 185 may processoperating data for a time period during which an incident has occurredto reconstruct the incident and obtain relevant information, such aslocation, force of impact, indications that autonomous safety featureswere triggered, etc.

In further embodiments, the on-board computer 114, the smart homecontroller 185, and/or the personal electronic device 189 may testcomponents of the vehicle 108, the smart home 187, and/or the personalelectronic device 189 that may be affected to determine operatingstatus. For example, the on-board computer 114, the smart homecontroller 185, and/or the personal electronic device 189 may determinea tire of the vehicle 108 has gone flat based upon vehicle headingrelative to control data from one or more autonomous operation featuresin response to an indication from a tire pressure sensor that the tirepressure has dropped. In embodiments associated with the vehicle 108,the degree of divergence between the control commands and the observedvehicle trajectory may be further used to determine the urgency ofrepairing or replacing the tire. As another example, the on-boardcomputer 114, the smart home controller 185, and/or the personalelectronic device 189 may determine whether an incident has occurredbased upon operating data (such as sensor data from accelerometers) and,if so, whether the incident requires a response.

At block 508, the on-board computer 114, the smart home controller 185,and/or the personal electronic device 189 may determine whether anincident, including a cyber-attack, has been determined to haveoccurred. As discussed above, the on-board computer 114, the smart homecontroller 185, and/or the personal electronic device 189 may ignoreincidents not requiring a response. If no incident is determined to haveoccurred (or if no response is required), the method 500 may continue bydetermining whether to continue monitoring (block 516), as discussedbelow. If at least one incident requiring a response is determined tohave occurred, the method 500 may continue by determining damageassociated with the incident (block 510).

At block 510, the on-board computer 114, the smart home controller 185,and/or the personal electronic device 189 may determine damageassociated with the determined incident. Such damage determination mayinclude assessing or estimating damage to the vehicle 108, the smarthome 187, and/or the personal electronic device 189 or another vehicle,another smart home, an injury to a pedestrian or passenger, or damage toanother object in the autonomous operating environment. The damage mayinclude physical damage to the sensors 120 or other components of thevehicle 108, the smart home controller 185, and/or the personalelectronic device 189, such as the sort of damage that typically occursduring collisions and/or other loss-events. The damage may likewiseinclude electronic damage to software involved in autonomous operation,such as the sort of damage that typically results from unauthorizedaccess to a computer system or infection of a computer system bymalicious code, such as when the component is the target of acyber-attack. The on-board computer 114, the smart home controller 185,and/or the personal electronic device 189 may determine the damage basedupon the obtained operating data and/or other data described elsewhereherein, which may include data from one or more sensors 120. In someembodiments, determining damage may include determining whether thevehicle 108, the smart home controller 185, and/or the personalelectronic device 189 can continue to operate in a fully or partiallyautonomous operation mode within predetermined safety parameters and/orwhether a passenger and/or occupant is capable of operating the vehicle108, the smart home 187, and/or the personal electronic device 189 in amanually and/or semi-autonomously within predetermined safety parameters(i.e., having risk levels for such operation below predetermined safeoperation threshold levels of risk).

Determining damage to components of the vehicle 108, the smart home 187,and/or the personal electronic device 189 may include determining thatsensors 120 are not functioning properly based upon conflicting sensorreadings, error signals, and/or sensor unresponsiveness. In someembodiments, multiple sensor failures in a region of the vehicle 108,the smart home 187, and/or the personal electronic device 189 may beused to determine damage to other components (e.g., body or structuraldamage in the area of the sensors). For example, the on-board computer114 may determine that damage to multiple sensors 120 in the front-rightportion of the vehicle 108 following a collision further indicates thatheadlights, signal lights, and the front bumper in that area are likelyalso damaged. As another example, the smart home controller 185 maydetermine damage to the basement of the smart home 187 further indicatesthat electronic devices in the basement are likely also damaged. Infurther embodiments, operating data regarding the vehicle 108, the smarthome 187, and/or the personal electronic device 189 (such as dataindicating the vehicle's movement or the location of other objects inthe autonomous operating environment) may be used by the on-boardcomputer 114, the smart home controller 185, and/or the personalelectronic device 189 to determine damage to the vehicle 108, the smarthome 187, and/or the personal electronic device 189, damage to othervehicles and/or smart homes, damage to other objects, or injuries topersons in the operating environment. Such operating data may includetelematics data regarding vehicle movement, position, direction, and/orspeed, as well as data regarding impact location and/or force.

At block 512, the on-board computer 114, the smart home controller 185,and/or the personal electronic device 189 may determine a response tothe incident. Such response may be determined based upon the type ofincident and/or the damage involved. Additional factors may also be usedto determine the appropriate response to the incident, such as location,urgency of an injury, importance of an interrupted vehicle trip,availability of a vehicle operator to reach the vehicle location, safetyconsiderations, legal obligations regarding the incident, or otherfactors. The on-board computer 114, the smart home controller 185,and/or the personal electronic device 189 may select from a plurality ofpotential responses, ranging from completely shutting down to continuingto operate in a fully autonomous mode along the remainder of anunmodified route to a destination without any changes. In someembodiments, the determined response may include one or morenotifications to an interested party remote from the vehicle 108 and/orthe smart home 185, such as an owner, occupant, operator, or insurer ofthe vehicle 108, the smart home 187, and/or the personal electronicdevice 189. Such notifications may be generated by the on-board computer114, the smart home controller 185, and/or the personal electronicdevice 189 or by the server 140 based upon information received from theon-board computer 114, the smart home controller 185, and/or thepersonal electronic device 189 and may be transmitted via the network130.

In some embodiments, the response may further include notifications toone or more parties associated with another object involved in theincident, such as an owner, occupant, or insurer of another vehicleand/or smart home. Such notices may be communicated directly usingcommunication components of the vehicle 108, the smart home controller185, and/or the personal electronic device 189 or may be communicatedvia the network 130 when a relevant party is remote from the site of theincident. For example, a notification to a utility company may be sentin response to determining that the vehicle 108 has collided with autility pole or other infrastructure (or utility infrastructure hascollided with the vehicle 108) and/or in response to determining thatthe utility has been cut off at the smart home 187, which may requireinspection and/or repair.

In situations in which the determined damage has significantly impairedthe ability of the vehicle 108, the smart home 187, and/or the personalelectronic device 189 to operate safely in a fully autonomous operationmode, the determined response may include a determination not tocontinue operation or to cease autonomous operation of the vehicle 108,the smart home 187, and/or the personal electronic device 189. In someembodiments, the vehicle 108, the smart home 187, and/or the personalelectronic device 189 may be completely inoperable, and the response mayinclude automatically contacting a third party to have the vehicle towedto a service or disposal facility and/or have the smart home sitecleared for rebuilding. In these embodiments, the response mayadditionally include contacting an autonomous vehicle dispatch center todispatch a replacement autonomous vehicle to the location of the vehicle108. If the vehicle 108 is capable of being operated at least shortdistances, the on-board computer 114 may identify a position out of theflow of traffic to which to move the vehicle 108. In such instances, theon-board computer 114 may determine a response including moving thevehicle 108 out of a traffic lane to a nearby location, such as aroadway shoulder, a parking lane, or a parking lot. In some embodiments,the response may include such movement regardless of whether the vehicle108 is able to safely complete the original vehicle trip, such as insituations in which the on-board computer 114 has determined the vehicle108 should remain at the incident location.

The response may include causing the vehicle 108 to remain at the siteof the incident for a number of reasons, including legal obligations,further assessment of the incident, further analysis of the functionalstate of the vehicle 108, or communicating with or aiding another partyinvolved in the incident. Thus, some embodiments may include determininga response that includes establishing communication between persons inthe vicinity of the vehicle 108, the smart home 187, and/or the personalelectronic device 189 and a remote owner, operator, or other partyassociated with the vehicle 108, the smart home 187, and/or the personalelectronic device 189 via the network 130. Such communication may beestablished using one or more speakers, microphones, cameras or imagecapture devices, displays 202, or other components of the vehicle 108,the smart home 187, and/or the personal electronic device 189 tofacilitate two-way communication. In this manner, a remote owner oragent may communicate with persons at the location of the incident, suchas police, paramedics, operators of other vehicles, pedestrians, etc.This communication may be necessary to provide or exchange informationregarding the vehicle 108, the smart home 187, and/or the personalelectronic device 189 or the incident, or for emergency responsecoordination.

In some embodiments, the response may include an emergency action and/orresponse. Such emergency action and/or response may includeautomatically communicating with an emergency response service to obtainemergency medical, fire, or police assistance, which may includecommunication via the server 140 or via a remote agent contacting anappropriate emergency responder. In response to determining that anincident has resulted in a serious injury to a pedestrian, an occupant,or a passenger of another vehicle, for example, the on-board computer114, the smart home controller 185, and/or the personal electronicdevice 189 may determine an emergency action and/or response thatincludes communication to request emergency assistance from an emergencyresponse service. The emergency action and/or response may furtherinclude establishing communication with the injured person or otherpersons in the vicinity of the vehicle 108, the smart home 187, and/orthe personal electronic device 189, as discussed above.

Such emergency communication may be established between an emergencyassistance representative (e.g., a representative of the owner,occupant, operator, insurer, or other interested party or arepresentative of an emergency response service) and the injured personor other persons in the vicinity of the vehicle 108, the smart home 187,and/or the personal electronic device 189 using communication componentsof the vehicle 108, the smart home 187, and/or the personal electronicdevice 189. Emergency actions and/or responses may additionally, oralternatively, include use of the vehicle 108 to transport one or morepersons from the location of the incident to an emergency or otherappropriate facility. If the vehicle 108 is determined to be operablewith risk levels below a safe operation threshold, the emergency actionand/or response may include facilitating access to the vehicle 108 andusing the vehicle 108 to transport the one or more persons to theemergency facility (such as a hospital) in a fully autonomous mode.

In some embodiments, the response may include granting limited access tooperate the vehicle 108, the smart home 187, and/or the personalelectronic device 189 to other persons. Such access may likewise begranted to allow manual or autonomous operation of the vehicle 108, thesmart home 187, and/or the personal electronic device 189, such as foremergency transportation, to move the vehicle 108 out of the path oftraffic, and/or to enable fire equipment at the smart homer 187. Forexample, access may be granted to allow the vehicle 108 to be moved to ashoulder of a road, an emergency stopping area, or a nearby parking lot.The response may thus include unlocking doors and allowing control forvehicle and/or smart home operation up to a threshold (e.g., a distancethreshold such as one mile, a time threshold such as ten minutes, etc.).The access granted may include manual access to allow a vehicle operatorto drive the vehicle 108 or autonomous access to allow a local or remotevehicle operator to direct the vehicle 108 to a user-selected location.In some embodiments, such access may only be granted to authorizedemergency personnel, such as police, fire, or medical personnel. Thislimited access may be achieved by wireless communication of an officialverification signal to the vehicle 108, the smart home 187, and/or thepersonal electronic device 189 via the communication component 122.Similarly, a remote vehicle operator at an emergency response facilitymay be granted control over the vehicle 108 to control the vehicle'smovements in an autonomous mode from a remote location, such as byselecting a parking location out of the flow of traffic (e.g., along ashoulder of a road).

In further embodiments, the on-board computer 114, the smart homecontroller 185, and/or the personal electronic device 189 may determinethat the vehicle 108, the smart home 187, and/or the personal electronicdevice 189 should be repaired. For example, one or more sensors 120 ofthe vehicle 108, the smart home 187, and/or the personal electronicdevice 189 may be malfunctioning, necessitating repair or replacement.In instances in which the vehicle 108 is capable of continued safeoperation in a fully autonomous mode, the response may include selectinga repair facility and routing the vehicle 108 to the repair facility byfully autonomous operation to be repaired. The selection of the repairfacility may include a determination that the repair facility is capableof providing the necessary repairs, has a sufficient stock of componentsdetermined to be required, and/or is able to perform the repairs withintiming and budgetary constraints. Selection may further includecommunicating with the repair facility (either automatically by theon-board computer 114, the smart home controller 185, and/or thepersonal electronic device 189, or server 140 or manually by a personassociated with the vehicle 108, the smart home 187, and/or the personalelectronic device 189) to schedule or confirm an appointment for therepairs.

In some embodiments, a determination regarding liability or insurancecoverage for the costs associated with the repairs may be made, and therelevant payor may be required to authorize the repairs. In furtherembodiments, alternative transportation and/or lodging may beautomatically arranged for a vehicle and/or smart home owner or operatorwhile the vehicle 108 and/or smart home 187 is undergoing repair. Saidalternative transportation may include taxi service, temporaryvehicle-sharing membership, vehicle rental, or similar temporaryreplacement transportation. Similarly, alternative lodging may include ahotel, a time-share, a peer-to-peer lodging, a rental, or othertemporary lodging services.

In yet further embodiments, the response may include automatically andpermanently replacing the vehicle 108 with an equivalent vehicle. Anequivalent replacement vehicle may be one of equivalent make, model,year, style, color, mileage, age, equipment, components, or similarcharacteristics. Information regarding the exchange of the vehicles maybe automatically provided to vehicle owners, insurers, lienholders,government or private registries, or other relevant individuals ororganizations. Additionally, personal items within the vehicle 108 maybe transferred to the replacement vehicle, and settings andconfigurations may be transferred by electronic communication to thereplacement vehicle. In this manner, the exchange of the vehicle 108 forthe replacement vehicle may not require any action by an owner oroperator of the vehicle 108. In some embodiments, however, the owner mayconfirm or verify authorization to exchange the vehicle 108 for thereplacement vehicle.

At block 514, the one or more responses may be implemented. The on-boardcomputer 114, the smart home controller 185, and/or the personalelectronic device 189 may control the vehicle 108, the smart home 187,and/or the personal electronic device 189 to take actions necessary toimplement the determined responses, including controlling movement ofthe vehicle 108, controlling components of the vehicle 108, the smarthome 187, and/or the personal electronic device 189 to facilitatecommunication, controlling components of the vehicle 108, the smart home187, and/or the personal electronic device 189 to obtain additionalinformation or take actions, enabling or disabling motive functionalityof the vehicle 108, the smart home 187, and/or the personal electronicdevice 189, and/or shutting down the vehicle 108, the smart home 187,and/or the personal electronic device 189. In some embodiments, this mayinclude causing the vehicle 108 to operate in a fully autonomous manneralong a route to a destination, which may be the original destination ora new destination (e.g., an emergency facility, a repair facility,etc.). If the vehicle 108 is routed to a new destination, it maythereafter further be routed from the new destination to the originaldestination (such as after completion of repairs at a repair facility).In some embodiments, part of the implementation of the determined one ormore responses may be implemented by the server 140 or another deviceremote from the vehicle 108.

At block 516, the on-board computer 114, the smart home controller 185,and/or the personal electronic device 189 may determine whether tocontinue monitoring the vehicle 108, the smart home 187, and/or thepersonal electronic device 189 following the commencement or completionof implementation of the response. This determination may includedetermining whether the vehicle 108, the smart home 187, and/or thepersonal electronic device 189 is continuing to operate in a fullyautonomous mode. This determination may further include determiningwhether other factors indicate that monitoring should continue, such asa continued risk of further damage or additional incidents. For example,monitoring may continue when the vehicle 108 is stopped within a trafficlane or on a shoulder of a roadway because such position involves anincreased risk of further incidents, such as collisions with othervehicles. If operation or monitoring is determined to continue, themethod 500 may continue with the on-board computer 114, the smart homecontroller 185, and/or the personal electronic device 189 monitoringoperating data and/or other data for further indications of unusualconditions (block 502). Such monitoring may, of course, exclude knownindications of unusual conditions (e.g., known damaged or malfunctioningsensors). If monitoring is determined not to continue, the method 500may terminate.

In some embodiments, determination of incident occurrence (blocks 506and 508), damage assessment (block 510), and/or response determination(block 512) may involve a remotely located human reviewer. The reviewermay be an owner, occupant, operator, insurer, or agent associated withthe vehicle 108, the smart home 187, and/or the personal electronicdevice 189 or otherwise authorized to take action regarding the vehicle108, the smart home 187, and/or the personal electronic device 189. Suchreview may be used to verify or confirm assessments made by the on-boardcomputer 114, the smart home controller 185, and/or the personalelectronic device 189 or to make additional determinations where theon-board computer 114, the smart home controller 185, and/or thepersonal electronic device 189 cannot adequately assess the situation.

In some such embodiments, the on-board computer 114, the smart homecontroller 185, and/or the personal electronic device 189 may sendinformation via the network 130 to the remote reviewer, who may accessthe information via the network 130 or through the server 140. Suchinformation may include operating data (or a subset of relevantoperating data), images, or video recordings of the incident. In someembodiments, the remote reviewer may, with permission, operate one ormore cameras of the vehicle 108, the smart home 187, and/or the personalelectronic device 189 to obtain streaming video or periodic images fromthe vehicle 108, the smart home 187, and/or the personal electronicdevice 189 during review. When operating data is received from thevehicle 108, the smart home 187, and/or the personal electronic device189, the server 140 may further process the operating data to present itin a human-readable format (e.g., a table, chart, or graphicaldepiction) based upon calculations made from non-human-readableoperating data.

The remote reviewer may then verify, modify, or determine an occurrence,type of occurrence, and/or damage or severity of the occurrence basedupon the information received from the vehicle 108, the smart home 187,and/or the personal electronic device 189. This may includecommunicating with persons in the vicinity of the vehicle 108, the smarthome 187, and/or the personal electronic device 189 using thecommunication components of the vehicle 108, the smart home 187, and/orthe personal electronic device 189, as described above. The remotereviewer may further verify, modify, or determine one or more responsesto the incident and may cause one or more responses to be implemented bythe vehicle 108, the smart home controller 187, and/or the personalelectronic device 189.

Exemplary Salvage Assessment Methods

FIG. 6 illustrates a flow diagram of an exemplary salvage assessmentmethod 600 for automatically determining damage and/or salvage potentialfor components of an autonomous vehicle, smart home, and/or personalelectronic device. Such method 600 may be implemented following acollision or other loss-event associated with a vehicle 108, a smarthome 187, and/or a personal electronic device 189 to determine whichcomponents are damaged and which components may be salvaged for use asreplacement parts in other autonomous vehicles, smart homes, and/orpersonal electronic devices. For traditional environments withoutautonomous operation features, similar salvage assessment typicallyinvolves physical inspection of the damaged component. Unlike damage tocomponents of non-autonomous environments, damage to some components ofautonomous environments may not be evident to visible inspection. Evenwhen components of an autonomous environment appear to be functioning,signals or output from such components may be inaccurate due tounobserved damage. Therefore, the method 600 may be used to evaluatevehicle components of an autonomous vehicle, a smart home, and/or apersonal electronic device to determine salvage potential.

The exemplary salvage assessment method 600 may begin by determiningthat damage to a vehicle 108, a smart home 187, and/or a personalelectronic device 189 has occurred (block 602). Following suchdetermination, a salvage assessment device may be connected to thevehicle 108, the smart home 187, and/or the personal electronic device189 to evaluate component salvage potential (block 604), and one or morecomponents of the vehicle 108, the smart home 187, and/or the personalelectronic device 189 may be selected for salvageability assessment(block 606). For each selected component, the salvage assessment devicemay cause a test signal to be sent to the component (block 608), whichmay include sending the test signal through the on-board computer 114 ofthe vehicle 108, the smart home controller 185 of the smart home 187,and/or the personal electronic device 189.

The salvage assessment device may then detect or receive a response fromthe component (block 610), which may include detecting that thecomponent is unresponsive. An expected response may also be obtained bythe salvage assessment device (block 612), which may be compared againstthe received response (block 614). Based upon such comparison, thesalvage assessment device may then determine the salvage potential ofthe component (block 616). In some embodiments, the salvage assessmentdevice may further determine salvage potential for an additionalcomponent of the vehicle 108, the smart home 187, and/or the personalelectronic device 189 based upon the determined salvage potential of theone or more selected components (block 618).

At block 602, the method 600 may begin by determining that damage to thevehicle 108, the smart home 187, and/or the personal electronic device189 has occurred. Such determination may be made automatically by anon-board computer 114, the smart home controller 185, the personalelectronic device 189, and/or a server 140 based upon operating data orother information from the vehicle 108, the smart home 187, and/or thepersonal electronic device 189. For example, such determination may bemade following an indication of an unusual condition or loss-eventinvolving the vehicle 108, the smart home 187, and/or the personalelectronic device 189, as discussed elsewhere herein. Such determinationmay, alternatively, be received from an owner, occupant, operator, orother interested party. In some embodiments, the determination mayinclude determining that the vehicle 108, the smart home 187, and/or thepersonal electronic device 189 is sufficiently damaged that it requiresextensive repair or is irreparably damaged. In further embodiments, anadditional determination may be made that the vehicle 108, the smarthome 187, and/or the personal electronic device 189 is sufficientlydamaged that the cost of repair would exceed the value of the vehicle108, the smart home 187, and/or the personal electronic device 189repaired after being repaired, or that the vehicle, smart home, orpersonal electronic device is a total loss.

At block 604, the method 600 may continue with the connection of asalvage assessment device to the vehicle 108, the smart home 187, and/orthe personal electronic device 189. This may include connecting thesalvage assessment device to an on-board computer 114 of the vehicle108, the smart home controller 185 of the smart home 187, and/or thepersonal electronic device 189, or it may include bypassing the on-boardcomputer 114 and/or the smart home controller 185, to directly assessthe other components of the vehicle 108 and/or the smart home 187.Bypassing the on-board computer 114 and/or the smart home controller 185may be beneficial when the on-board computer 114 and/or the smart homecontroller 185 is or may be malfunctioning. In further embodiments, thesalvage assessment device may preliminarily evaluate the operation ofthe on-board computer 114 and/or the smart home controller 185, thendetermine whether to connect to the components through the on-boardcomputer 114 or to bypass the on-board computer 114 and/or the smarthome controller 185 based upon the results of preliminary evaluation. Insuch embodiments, the salvage assessment device may be configured topresent test commands to the on-board computer 114 and/or the smart homecontroller 185 or may cause the on-board computer 114 and/or the smarthome controller 185 to run one or more self-diagnostic routines. In someembodiments, the salvage assessment device may connect to the on-boardcomputer 114 and/or the smart home controller 185 to control theon-board computer 114 and/or the smart home controller 185 to generate,transmit, and/or receive signals related to assessing components of thevehicle 108, the smart home 187, and/or the personal electronic device189. In further embodiments, the on-board computer 114 and/or the smarthome controller 185 may be used as the salvage assessment device, inwhich case no additional connection may be required.

The salvage assessment device may be a mobile device 110, as describedelsewhere herein, which may be a special-purpose computing device or ageneral purpose computing device (e.g., a smartphone or tabletcomputer). The salvage assessment device may include or be connected toa special-purpose connector configured to connect to a communicationport of the vehicle 108, the smart home controller 185, and/or thepersonal electronic device 189. Such communication port may be anon-board diagnostic (OBD) port, such as an OBD-II or EOBD port, auniversal serial bus (USB) port, an Ethernet port, or other ports thatsupport the interconnection between two electronic devices. In furtherembodiments, the salvage assessment device may be configured to connectto the on-board computer 114, the smart home controller 185, and/or thepersonal electronic device 189 wirelessly via a WiFi, Bluetooth, orother wireless electronic communications. Establishing the wired orwireless communication connection between the salvage assessment deviceand the vehicle 108, the smart home 187, and/or the personal electronicdevice 189 may include causing the on-board computer 114, the smart homecontroller 185, and/or the personal electronic device 189 and/or othercomponents of the vehicle 108, the smart home 187, and/or the personalelectronic device 189 to enter into a diagnostic mode for evaluation.

At block 606, the salvage assessment device may determine one or morecomponents of the vehicle 108, the smart home 187, and/or the personalelectronic device 189 to assess. Such components may include sensors120, part or all of the on-board computer 114, the smart home controller185, and/or the personal electronic device 189, and/or separateautonomous operation feature components. Determining the one or morecomponents of the vehicle 108, the smart home 187, and/or the personalelectronic device 189 to assess may include selecting the componentsfrom a plurality of components of the vehicle 108. To this end, thesalvage assessment device may receive information regarding theplurality of components of the vehicle 108, the smart home 187, and/orthe personal electronic device 189, which may include accessing suchinformation from a program memory 208 or a database 146. The salvageassessment device may then identify one or more components of thevehicle 108, the smart home 187, and/or the personal electronic device189 for evaluation. Such identification may include selecting the one ormore components based upon operating data received from the vehicle 108,the smart home 187, and/or the personal electronic device 189, whichoperating data may be associated with a collision or other loss-eventresulting in damage to the vehicle 108, the smart home 187, and/or thepersonal electronic device 189. The salvage assessment device maydetermine that some components of the vehicle 108, the smart home 187,and/or the personal electronic device 189 should or should not beevaluated because of high probabilities that such components either areor are not damaged based upon the operating data. For example, in ahigh-speed head-on collision, the salvage assessment device maydetermine that sensors located in the front bumper of the vehicle 108are highly likely to be damaged, therefore determining not to selectsuch sensors for evaluation. As another example, in a tree-fall event,the salvage assessment device may determine that sensors located on theroof of the smart home 187 are highly likely to be damaged, andsimilarly are determined not to be selected for evaluation. In furtherembodiments, the salvage assessment device may iteratively evaluate andassess all components capable of electronic communication with thesalvage assessment device that can be identified as being disposedwithin the vehicle 108, the smart home 187, and/or the personalelectronic device 189.

At block 608, the salvage assessment device may cause one or more testsignals to be sent to each of the determined components. The salvageassessment device may generate and communicate such test signals to thecomponents, or the salvage assessment device may control the on-boardcomputer 114, the smart home controller 185, and/or the personalelectronic device 189 to generate and/or communicate the test signals tothe components. The test signals may cause the components to return oneor more response signals to the on-board computer 114, the smart homecontroller 185, and/or the personal electronic device 189 or the salvageassessment device. Such response signals may indicate a self-assessmentof the component, an acknowledgement of receipt of the test signal bythe component, a value measured or determined by the component, oranother response by the component upon receipt of the test signal.

At block 610, the salvage assessment device may receive the one or moreresponses from the one or more components. The responses may be receivedvia the on-board computer 114, the smart home controller 185, and/or thepersonal electronic device 189 in some embodiments. The responses mayinclude response signals from the components or information basedthereon from the on-board computer 114, the smart home controller 185,and/or the personal electronic device 189. In some embodiments, theresponses may include an implied response indicating componentdisconnection or malfunctioning (e.g., as a result of damage), which maybe inferred by the salvage assessment device or on-board computer 114,the smart home controller 185, and/or the personal electronic device 189from an absence of a response signal within a usual time period forresponse from the component. Such received or implied responses may thenbe used to determine damage or salvageability of components of thevehicle 108, the smart home 187, and/or the personal electronic device189.

At block 612, the salvage assessment device may obtain one or moreexpected responses for the one or more components. Such expectedresponses may be indicative of ordinary or usual responses of the one ormore components to the one or more test signals, and the expectedresponses may be obtained from a program memory 208 or a database 146.The expected responses may include ranges of response signals associatedwith proper operation of components, such as ranges of sensor datagenerated by a sensor 120 when functioning properly.

In some embodiments, the expected responses may be based at least inpart upon operating data received from additional components of thevehicle 108, the smart home 187, and/or the personal electronic device189. For example, sensor data from other sensors 120 may be used todetermine an expected response from a component to be evaluated. Infurther embodiments, such operating data may be received from othercomponents determined to be operating properly or may be received from aplurality of other components of unknown status, in which latter casethe expected responses may include a plurality of ranges based uponwhether the other components are functioning properly or aremalfunctioning. In yet further embodiments, known characteristics of theautonomous environment at the time of assessment (e.g., distance fromobjects near the vehicle 108, the smart home 187, and/or the personalelectronic device 189) may be used to determine expected responses ofthe one or more components.

At block 614, the salvage assessment device may compare the receivedresponses and the expected responses for the one or more components toevaluate the operating status or condition of the one or morecomponents. This may include determining whether a received response iswithin a range of acceptable responses based upon one or more associatedexpected responses indicative of proper functioning of the component. Insome embodiments, this may include comparing received responses andexpected responses for a plurality of components to determine whetherthe received responses are consistent with other received responses,particularly with those received responses that are consistent with theexpected responses.

At block 616, the salvage assessment device may then determine a salvagepotential indicative of whether each of the one or more components issalvageable based upon such comparisons between received and expectedresponses. The salvage potential of a component may be associated withan estimate of damage, which may include an estimate of a level, type,or extent of damage. For example, a component may be determined to havesuffered minor damage based upon a shift in an average value of the oneor more response signals associated with the component, even though theresponses are generally within an acceptable range based upon theexpected responses associated with the component. A component determinedto be damaged may be further determined not to be salvageable or may bedetermined to be partially salvageable. In instances in which multipleresponse signals are associated with a component (e.g., where acomponent includes multiple sensors 120), the component may bedetermined to be partially malfunctioning and partially operational dueto subcomponent damage. In some embodiments, such situations may furtherbe evaluated by the salvage assessment device to determine whether thesubcomponents of the component may be repaired or replaced, which may befurther used to determine whether the component is salvageable.

At block 618, in some embodiments, the salvage assessment device mayfurther determine a salvage potential of one or more additionalcomponents of the vehicle 108, the smart home 187, and/or the personalelectronic device 189 based upon the determined salvage potential ordamage associated with one or more components. The additional componentsmay be disposed within the vehicle 108, the smart home 187, and/or thepersonal electronic device 189 in physical proximity to a set of the oneor more components, such that the additional components may be expectedto have suffered similar damage levels as the set of evaluatedcomponents. In some embodiments, the additional components may includecomponents that are not sensors or autonomous operation features.

Such additional components may not be configured for electroniccommunication with the on-board computer 114, the smart home controller185, and/or the personal electronic device 189, even if the additionalcomponents are controlled by the on-board computer 114, the smart homecontroller 185, and/or the personal electronic device 189. For example,the additional components may include headlights, signal lights, bodypanels, roofing, siding, support beams, and/or other structural ornon-communicating parts of the vehicle 108, the smart home 187, and/orthe personal electronic device 189. Determining the salvage potential ofsuch additional components may include determining an estimated level ofdamage for an area of the vehicle 108, the smart home 187, and/or thepersonal electronic device 189 associated with an additional componentbased upon estimated levels and/or types of damages associated with theset of components.

In some embodiments, the salvage assessment device may furthercommunicate the determined salvage potential of the components and/oradditional components to a server 140 for storage in a database 146 orto a mobile device 110 for storage in a program memory 208. The salvageassessment device may additionally, or alternatively, presentinformation indicating the salvage potential of the components to a userof the salvage assessment device. This may include a report summarizingthe salvage potential of the vehicle 108, the smart home 187, and/or thepersonal electronic device 189 and/or the components thereof. In someembodiments, information regarding costs or values associated with thecomponents may be used to estimate salvage values of one or morecomponents or of the vehicle 108, the smart home 187, and/or thepersonal electronic device 189.

Exemplary Malfunction Detection Methods

FIG. 7 illustrates a flow diagram of an exemplary malfunction detectionmethod 700 for identifying malfunctioning sensors of an autonomousvehicle 108 and/or a smart home 187. Such method 700 may be used todetect sensors of an autonomous vehicle and/or smart home that aremalfunctioning and, in some instances, to correct the malfunction. Themethod 700 may be implemented during operation to ensure the vehicle 108and/or the smart home 187 is functioning properly and to detectmalfunctions caused by damage, environmental conditions, or othercauses. Because autonomous vehicles rely heavily on accurate sensor datato make control decisions to operate the vehicle, damaged or otherwisemalfunctioning sensors may pose a hazard to the vehicle, other vehicles,passengers, or pedestrian in the vehicle's operating environment.Similarly, smart homes rely heavily on accurate sensor data (e.g., hometelematics data) to ensure proper operation of smart equipment and/or todetect emergencies that threaten damage to the smart home, damaged orotherwise malfunctioning sensors may lead to improper reporting ofconditions pose hazards to occupants, passersby, and/or other property.Such sensors may malfunction for various reasons, includingmanufacturing defects, ordinary wear, collision damage, cyber-attacks,or weather damage. In many cases, such damaged or otherwisemalfunctioning sensors may appear to be operating properly to both avehicle operator and to an on-board computer 114 of the vehicle 108and/or a smart home controller 185 of the smart home 187. The method 700evaluates the sensors to detect malfunctions and take actions to addressthe hazard posed by such malfunctions. Although the method 700 isdescribed below as being performed by the on-board computer 114 and/orthe smart home controller 187, for simplicity, it should be understoodthat one or more mobile devices 110 or servers 140 may alternatively, oradditionally, be used to perform part or all of the process.

The exemplary malfunction detection method 700 may begin by receivingsensor data from one or more sensors 120 of the vehicle 108, sensorsinterconnected with the smart home controller 187, and/or a personalelectronic device 189 (block 702). The method 700 may then iterativelyevaluate the sensors 120 to identify and respond to sensor malfunctions.A sensor is selected for evaluation (block 704), and one or more signalsassociated with the selected sensor are obtained (block 706). A range ofsignal values associated with proper functioning of the sensor may bedetermined (block 708) and compared against the one or more signalsassociated with the selected sensor to determine whether the sensor ismalfunctioning (block 710). If the sensor is determined to bemalfunctioning (block 712), a cause of the malfunction and/or otherinformation associated with the malfunction may be determined (block714). Based upon the determined information, one or more responses tothe malfunction may then be determined (block 716) and implemented(block 718). If another sensor is to be evaluated (block 720), themethod 700 selects and evaluates another sensor. If no sensors remain tobe evaluated (block 720), the method 700 may terminate.

At block 702, the on-board computer 114 may receive sensor data from oneor more sensors 120 of the vehicle 108, sensors interconnected with thesmart home controller 187, and/or a personal electronic device 189. Thesensor data may include a plurality of signals associated with the oneor more sensors 120, which signals may be generated during operation ofthe vehicle 108, operation of the smart home 187, and/or the personalelectronic device 189. The sensor data may be received when the vehicle108 is first started, when the vehicle 108 is shut down, when anindividual enters the smart home 187, when an individual egresses thesmart home 187, or upon the occurrence of an event. Such events mayinclude a user-initiated request to evaluate the sensors 120 ordetection of an indication of an unusual condition, such as a collisioninvolving the vehicle 108, damage to the smart home 187, and/or anindividual experiencing a medical emergency. In other embodiments,however, the sensor data may be received and evaluated to determinewhether a sensor 120 is malfunctioning without any indication of acollision and/or other event. In particular embodiments, the sensor datamay be received continuously or periodically during operation of thevehicle 108 by the on-board computer 114 and/or during operation of thesmart home 187 by the smart home controller 185. The on-board computer114 and/or smart home controller may then process or store the sensordata for future processing.

In some embodiments, the sensor data may additionally or alternativelyinclude sensor data received from a sensor not included in the one ormore sensors 120 of the vehicle 108 and/or or one or more sensorslocated on or proximate to the smart home 187. As an example, a sensorof the one or more smart infrastructure components 188 may transmitsensor data to the on-board computer 114 via the network 130. As anotherexample, the sensor data may include sensor data received from thepersonal electronic device 189.

At block 704, the on-board computer 114 and/or smart home controller 185may select a sensor from the one or more sensors 120, the one or moresensors located on or proximate to the smart home 187, the sensors ofthe one or more smart infrastructure components 188, and/or the personalelectronic device 189 to evaluate. The sensor may be selected based uponan indication of potential malfunction, such as inconsistent or unusualsensor data from the selected sensor. In some scenarios, theinconsistent or unusual sensor data may be indicative of a cyber-attackdirected at the vehicle 108, smart home 185, personal electronic device189, and/or the one or more smart infrastructure components 188. Whenthe method 700 is implemented in response to a collision and/or otherevent, the sensor may be selected based at least in part upon alikelihood of damage to the sensor from the collision and/or otherevent. In further embodiments, the sensor may be selected based upon ahierarchy of sensors 120, such that higher-level sensors may beevaluated first. If a higher-level sensor is determined to be operatingproperly, there may be no need to test lower-level sensors associatedwith the higher-level sensor. This may reduce the processing timerequirements for evaluating the sensors 120, thus enabling more frequentsensor evaluation (particularly during operation of the vehicle 108and/or occupancy of the smart home 187).

At block 706, the on-board computer 114 and/or smart home controller 185may obtain sensor data associated with the selected sensor. Such sensordata may include a set of signals from the sensor, such as signalsgenerated by the sensor during operation of the vehicle 108 and/or smarthome 187. The set of signals may include raw signal data from the sensoror signal data preprocessed by the sensor or by the on-board computer114 and/or smart home controller 185. The signal data may includediscrete data points generated by the sensor or samples of continuousdata generated by the sensor. The set of signals may be obtained over aperiod of time from the sensor, or the set of signals may be accessedfrom sensor data previously stored in a program memory 208. In someembodiments, the set of signals associated with the sensor may includeindications of sensor unresponsiveness, viz. indications that sensordata was not received from the sensor at times when sensor data wasrequested of the sensor or expected from the sensor.

At block 708, the on-board computer 114 and/or smart home controller 185may determine a range of signals associated with proper functioning ofthe sensor. The range of signals may include a range of values for thesignals that correspond to proper functioning of the sensor.Alternatively, a range of signal values associated with a malfunctioningstatus of the sensor may be determined. The range of signal valuesassociated with proper functioning of the sensor may be determined basedupon specifications for the sensor, historical data from the sensor, orestimates of what the signal values should be based upon contemporaneoussensor data from other sensors of the vehicle 108 and/or smart home 187.The range of signal values may thus be determined based upon a secondset of signals from the sensors 120 of the vehicle 108, the one or moresensors located on or proximate to the smart home 187, the sensors ofthe one or more smart infrastructure components 188, and/or the personalelectronic device 189.

The second set of signals may include a plurality of signals previouslyreceived from the sensor during a plurality of separate previous vehicletrips of the vehicle 108 and/or prior operation of the smart home 187.The second set of signals may, additionally or alternatively, include aplurality of additional signals from one or more additional sensors 120of the vehicle 108, the one or more sensors located on or proximate tothe smart home 187, the sensors of the one or more smart infrastructurecomponents 188, and/or the personal electronic device 189 other than theselected sensor. The second set of signals may be used to estimate oneor more ranges of expected responses of the sensor to various conditionsin the vehicle and/or home operating environment. This information maybe further used to estimate a range of values for an expected responsesignal of the sensor based upon concurrent sensor data from the one ormore additional sensors. For example, the additional sensors mayindicate an obstruction approximate ten feet ahead of the vehicle 108,in which case the determined range of signal values for the sensor maybe associated with detection of an object between nine and eleven feetahead of the vehicle 108. As another example, the additional sensors mayindicate a temperature in a room proximate to the room of the smart home187 in which the selected sensor is disposed, in which case, thedetermined range of signal values may be a threshold variance from thevalue detected by the additional sensors.

At block 710, the on-board computer 114 and/or smart home controller 185may compare the set of signals obtained from the selected sensor againstthe determined range of signals associated with proper functioning ofthe sensor. The sensor may thus be determined to be properly functioningwhen the signal values of the set of signals are within the range ofsignal values associated with proper functioning, or the sensor may bedetermined to be malfunctioning when the signal values of the set ofsignals are outside the range of signal values associated with properfunctioning. In some embodiments, the range of signal values associatedwith proper functioning may exclude values associated with indicationsthat the sensor is unresponsive or that sensor data from the sensor isunavailable, in which case signals from the sensor indicating that thesensor is unresponsive may be outside the range of determined signalvalues associated with proper functioning. In embodiments in which therange of signals associated with proper functioning is determined usinga second set of signals from additional sensors, the comparison mayinclude determining whether the contemporary signals in the set ofsignals and the second set of signals are consistent or inconsistent.When inconsistencies are determined to exist, the sensor may bedetermined to be malfunctioning.

At block 712, the on-board computer 114 and/or smart home controller 185may then determine whether the sensor is malfunctioning based upon thecomparison between the set of signals obtained from the sensor and thedetermined range of signal values associated with proper functioning ofthe sensor. In some embodiments, determining that the sensor ismalfunctioning based upon the comparison may include determining aprobability that the sensor is currently or will be malfunctioningwithin a predetermined future time period based upon the comparison ofsignal values.

The probability may indicate a likelihood that the sensor is currentlymalfunctioning based upon the number, frequency, or magnitude ofdeviations of the signals from the range of values associated withproper functioning. For example, an outlier signal value in the set ofsignals from the sensor may be associated with a lower probability ofmalfunction if it is 5% higher than an upper bound of the range ofsignal values than if the outlier is 25% higher than the upper bound.Similarly, a set of signals with one outlier may be associated with alower probability of malfunction than a set having the same number ofsignals with ten outliers. In some embodiments, the probability may beindicative of a prediction of future failure of the sensor, whichprediction may be informed by information regarding failure rates ofsimilar sensors from a plurality of other autonomous vehicles and/orsmart homes retrieved from the database 146.

When the sensor is determined to be malfunctioning, the method 700 maycontinue to determine and implement a response at blocks 714-718. Whenthe sensor is determined not to be malfunctioning, the method 700 maycontinue to determine whether any other sensors remain to be evaluatedat block 720.

At block 714, the on-board computer 114 may further determine a cause ofthe malfunction, an extent of the malfunction, or other informationassociated with the malfunction. This may include accessing informationregarding malfunction of similar sensors from a plurality of otherautonomous vehicles and/or smart homes retrieved from the database 146.For example, a malfunction in the one or more smart infrastructurecomponents 188 may be determined by comparing signals received by aplurality of vehicles from the one or more smart infrastructurecomponents 188. In further embodiments, this may include evaluatingoperating data (including sensor data) from the vehicle 108, the smarthome 187, and/or the personal electronic device 189. Such operating datamay be associated with the malfunctioning sensor and/or other sensors120 within the vehicle 108, smart home 187, and/or associated with thepersonal electronic device 189. In some embodiments, this may includeobtaining and processing time-stamped operating data from a plurality oftimes, which times may associated with multiple time frames (e.g., tripsor occupancy periods) or may be associated with different points withina time period associated with part of one time frame. For example, eachof the plurality of times may be periodic sample points at which theon-board computer 114 and/or the smart home controller 185 storesoperating data (such as the signals in the set of signals) duringoperation.

The on-board computer 114 and/or the smart home controller 185 mayanalyze the time-stamped operating data to identify an indication of anunusual condition associated with the malfunction, such as a collisionor other damage-causing event. In addition to collisions,weather-related events (e.g., frost, water intrusion, excessive heat,etc.), blockage (e.g., dirt, water, or salt build-up on the sensor), orother events may be determined as the cause of the sensor malfunction.Other causes may include age (e.g., years in service) of the sensor,manufacturer defect, improper installation, or inadequate maintenance,among other causes.

In addition to the cause of the sensor malfunction, or as an alternativethereto, the on-board computer 114 and/or the smart home controller 185may determine other information associated with the sensor malfunction.Such information may include an extent of the damage to the sensor orthe degree to which the malfunction results in inaccurate or unreliablesensor data. Such information may similarly include informationassociated with repair or replacement options or requirements, usualcost to repair or replace the sensor, other sensors that frequentlymalfunction at the same time, risk levels associated with operation ofthe vehicle 108 and/or the smart home 187 without the sensor (which mayinclude a plurality of risk levels associated with different levels orsettings used for autonomous or semi-autonomous vehicle operation), etc.

In some embodiments, determining the cause of the sensor malfunction mayinclude determining fault or liability for the sensor malfunction. Thismay include an apportionment of liability for a cost of repair orreplacement of the first sensor between one or more of: a manufacturerof the sensor, a manufacturer of the vehicle 108, a manufacturer of asmart equipment, a manufacturer of the personal electronic device 189,an installer of the sensor, an insurer of the vehicle 108, an insurer ofthe smart home 187, an owner of the vehicle 108, an owner of the smarthome 187, or an owner, operator, or insurer of another vehicle and/orsmart home. In further embodiments, determining the cause of the sensormalfunction may include determining insurance coverage for repair orreplacement of the sensor based upon the determined cause and/or fault.For example, sensor damage determined to be caused by weather (e.g.,freezing temperatures resulting in sensor failure) may be determined tobe covered under an insurance policy.

In some additional or alternative embodiments, determining the cause ofthe sensor malfunction may include determining a software versionassociated with the malfunctioning sensor. To this end, in acyber-attack scenario, the malfunction may be caused by the sensorhaving an outdated or corrupted software version that was exploited bythe cyber-attack. In another scenario, a software update may enableadditional functionality by the sensor and/or the vehicle 108. In thisscenario, the determination of the software version may restrict one ormore potential responses to the cyber-attack. In an embodiment, thesoftware on the sensor may be updated remotely to an updated version, oranti-virus software may be initiated on the sensor.

At block 716, the on-board computer 114 and/or the smart home controller185 may determine one or more responses to the sensor malfunction basedupon the determined cause and/or other information. The one or moreresponses may be selected or otherwise determined in order to addressthe malfunction by correcting the malfunction, changing operation of thevehicle 108 and/or the smart home 187 to reduce the impact of themalfunction, warning a vehicle owner or operator of the malfunction,updating a software associated with the malfunctioning sensor and/or thevehicle 108, dispatching a backup autonomous vehicle, or taking otheractions to improve operation of the vehicle 108 and/or smart home 187after identification of the malfunction. In some embodiments, part orall of the determination of the one or more responses may be performedby the server 140 based upon information received from the on-boardcomputer 114, the smart home controller 185, and/or the personalelectronic device 189 via the network 130.

In some embodiments, vehicle-mounted sensors may be able to determine oridentify a number and type of passengers, such as zero passengers, twopassengers, a child passenger, an elderly passenger, a handicapped orblind passenger, etc. Based upon the number and type passengers, theresponse determined 716 may be impacted. For instance, changingoperation of the vehicle with zero passengers may be easier than withseveral passengers.

In further embodiments, the one or more responses may be determinedbased upon additional information received by the on-board computer 114,the smart home controller 185 or server 140, which additionalinformation may include information regarding similar sensormalfunctions from a plurality of other vehicles and/or smart homes. Suchadditional information may be used to determine risks associated withoperation of the vehicle and/or smart home while the sensor ismalfunctioning, repairs typically required to correct the malfunction,or adjustments to vehicle operation to minimize the effect of themalfunction under various conditions.

An alert or warning to an operator and/or owner of the vehicle 108,and/or an occupant and/or owner of the smart home 187, may be generatedin order to notify relevant parties of the malfunction. Suchnotification may be applicable to situations in which other remedialactions may be taken, as well as to situation in which no immediateremedy is available. In some embodiments, additional information may beincluded as part of the notification, such as information regardingseverity of the malfunction, risks associated with operation of thevehicle 108 and/or the smart home 187 with the malfunction occurring,recommendations for correcting the malfunction, recommendations foradjusting vehicle and/or smart home operation until the malfunction iscorrected, or costs associated with correcting the malfunction oroperating the vehicle 108 and/or the smart home 187 without correctingthe malfunction.

The notification may include recommendations to be implemented orselected by an operator, occupant, and/or owner, such as recommendationsto take one or more of the following actions: repair the sensor, replacethe sensor, avoid using one or more autonomous operation features of thevehicle 108 and/or the smart home 187, or avoid using one or moresettings associated with the one or more autonomous operation features.The notifications may likewise include information regarding risks orcosts associated with operation of the vehicle 108 and/or the smart home187 without correcting the malfunction. This information may includeestimates of increased risks for each of various operation settings orlevels of autonomous operation feature usage. This information maysimilarly include adjustments to costs or coverage levels associatedwith an insurance policy for the vehicle 108 and/or the smart home 187based upon the sensor malfunction. Such adjustments may be immediate ormay be prospective (i.e., depending upon the actual usage of the vehicle108 and/or the smart home 187 following the notification).

The one or more responses may include recommending or enactinglimitations on use of one or more autonomous operation features orsettings. The limitations may be determined based upon risks associatedwith use of autonomous operation features or setting. Such risks may bedetermined by identifying one or more autonomous operation features orsettings of the vehicle 108 and/or the smart home 187 that utilizesensor data from the malfunctioning sensor, then determining one or morerisk levels associated with use of each such autonomous operationfeature or setting while the sensor is malfunctioning. Limitations onuse of the one or more autonomous operation features or settings may bedetermined for the operating environment of the vehicle 108 and/or thesmart home 187, in order to reduce risks associated with autonomous orsemi-autonomous operation (e.g., to reduce risk levels to below amaximum safe operation threshold level of risk). In some embodiments,determination of such limitations may include comparing risk levelsassociated with use of the autonomous operation features or settingswith risk levels associated with operation by a specific operator and/oroccupant without such autonomous operation features or settings.

Where the response includes recommending limited use of one or moreautonomous operation features or settings to a specific operator and/oroccupant, such recommendation may be conveyed together with anindication of a risk or cost associated with noncompliance with therecommendation. In other embodiments, the response may include enactingsuch limitations by limiting the operation of one or more autonomousoperation features or settings of the vehicle 108 and/or smart home 187,such as by disabling or locking relevant autonomous operation featuresor settings. Thus, the on-board computer 114 and/or the smart homecontroller 185 may disable or lock certain autonomous operation featuresor settings. In further embodiments, the response may include an optionallowing the operator and/or occupant to override such disabling orlocking of the autonomous operation features or settings.

The one or more responses may include repairing or replacing themalfunctioning sensor. This response may include a determination of oneor more repairs to be performed and/or one or more components to bereplaced. Such determination may further be based upon similar repairspreviously performed on other similar vehicles and/or smart homes tocorrect similar sensor malfunctions. In some embodiments, the responsemay include automatically scheduling repair or replacement of thesensor, which may include arranging for the sensor to be repaired orreplaced by a repair service provider, as discussed elsewhere herein. Infurther embodiments, liability or insurance coverage for such repairservice to correct the sensor malfunction may be determined, and theresponse may include automatically processing a payment for the service.In further embodiments, repairing the malfunctioning sensor may includeresetting, restarting, rebooting, recalibrating, or otherwise attemptingto clear the malfunction by returning the sensor to a predetermined ordefault state. This may be of particular use where a software error hascaused the malfunction, such that resetting the sensor (such as byrebooting the sensor) may correct the malfunction.

At block 718, the on-board computer 114 and/or the smart home controller185 may cause the one or more responses to be implemented.Implementation may include presentation of alerts or warnings to anoperator, owner, occupant and/or other interested party. Implementationmay likewise include disabling or locking autonomous operation featuresor settings thereof, which may be overridden by an operator and/oroccupant in some embodiments.

In further embodiments, implementation may include monitoring usage ofthe vehicle 108 and/or the smart home 187, such as by storing operatingdata. Usage may be monitored to determine whether the vehicle and/orsmart home is being operated in accordance with recommended usage levelsand settings for autonomous operation features impacted by the sensormalfunction, as well as to obtain additional information regarding thesensor. In yet further embodiments, such usage information may be usedto determine and/or implement an adjustment to a cost or coverageassociated with an insurance policy associated with the vehicle 108and/or the smart home 187, which adjustment may be based upon risklevels associated with the determined usage. Implementation may likewiseinclude scheduling repair or replacement of the malfunctioning sensor bya repair service provider, which may include automatically controllingthe vehicle 108 to travel to a repair location. In some embodiments, apayment for such repair or replacement service may be automatically madeor facilitated by the server 140.

At block 720, the on-board computer 114 and/or the smart home controller185 may determine whether to evaluate another sensor of the one or moresensors 120, the one or more sensors located on or proximate to thesmart home 187, the sensors of the one or more smart infrastructurecomponents 188, and/or the personal electronic device 189 (which mayalso determine a type and number of passengers in the vehicle 108 and/orthe smart home 187). When an additional sensor is identified forevaluation, the method 700 may continue with selecting another sensorfor evaluation (block 704). When no additional sensors are identifiedfor evaluation, the method 700 may terminate. Prior to termination insome embodiments, the on-board computer 114 and/or the smart homecontroller 185 may generate or store a summary report regarding thestatus of the sensors 120, the one or more sensors located on orproximate to the smart home 187, the sensors of the one or more smartinfrastructure components 188, and/or the personal electronic device189.

Exemplary Malfunction Assessment Methods

FIG. 8 illustrates a flow diagram of an exemplary malfunction assessmentmethod 800 for determining the impact of malfunctioning components onthe operation of an autonomous vehicle 108. Such method 800 may beuseful in determining not only the risk of accidents or other problemscaused by component failure within an autonomous vehicle, a smart homeor a personal electronic device, but also the extent or severity of suchproblems. By assessing both risks and results associated with autonomousoperation components, the reliability and robustness of the autonomoussystems may be determined. Based upon such determinations, somecomponents of the vehicle may be adjusted to reduce the likelihood ofsevere problems. This may be accomplished by automatically updating,upgrading, repairing, or replacing one or more components. Although themethod 800 is described below as being performed by a server 140 forsimplicity, it should be understood that one or more mobile devices 110,on-board computers 114, smart home controllers 185, and/or personalelectronic devices 189 may alternatively, or additionally, be used toperform part or all of the process.

The exemplary malfunction assessment method 800 may iteratively assessone or more autonomous operation components of the vehicle 108, thesmart home 187, and/or the personal electronic device 189 to determine alikelihood and impact of component malfunction. The method 800 may beginwith the selection of a component to assess (block 802). Operatinginformation may be obtained from a plurality of vehicles, smart homes,and/or personal electronic devices having instances of the component(block 804). Occurrences of component malfunctions may be identified inthe operating information (block 806), from which may be determinedrisks of malfunction (block 808), including risks of cyber-attack (block809), and results of such malfunctions (block 810).

The results of malfunctions may be associated with vehicle collisions orother hazardous events. A component profile may then be determinedand/or generated for the component based upon the determined risks andresults associated with component malfunction (block 812). A pluralityof components may be so assessed, in which case the preceding blocks802-812 may be repeated until no further components remain to beassessed (block 814). In some embodiments, the plurality of componentprofiles thus generated may be further used to determine and/or generatea risk profile for the vehicle 108, the smart home 187, and/or thepersonal electronic device 189 (block 816). In yet further embodiments,the method 800 may include determining one or more actions to reducerisk levels associated with the vehicle 108, the smart home 187, and/orthe personal electronic device 189 (block 818) and implement suchdetermined actions (block 820).

At block 802, the server 140 may select a component of a type used inautonomous operation of the vehicle 108. Such components may be limitedto autonomous operation component types of which the vehicle 108 has atleast one instance or copy installed. This selection may includeselecting components of particular interest, such as components newlyinstalled within the vehicle 108, the smart home 187, and/or thepersonal electronic device 189, or components recently updated orupgraded (e.g., by updating of software associated with a component to anew version). This selection may also include simply selecting eachknown component associated with autonomous operation of the vehicle 108,the smart home 187, and/or the personal electronic device 189, in turn.In some embodiments, this may include automatically identifying at theon-board computer 114, the smart home controller 185, and/or personalelectronic device 189 all autonomous operation components of the vehicle108, the smart home 187, and/or personal electronic device 189,respectively, such as by generating or processing a device registry ofthe components.

The components may include distinct autonomous operation features,hardware components associated therewith (e.g., sensors or controllers),versions of software programs for implementing part or all of anautonomous operation feature, version of operating systems forcontrolling autonomous operation of the vehicle 108, the smart home 187,and/or the personal electronic device 189, or types of sensorsconfigured to provide sensor data for autonomous or semi-autonomousoperation of the vehicle 108, the smart home 187, and/or the personalelectronic device 189. The component may be a general type of componentused in autonomous operation (e.g., a LIDAR unit) or may be a particulartype of a component (e.g., a specific model of a LIDAR unit produced bya particular manufacturer).

At block 804, the server 140 may obtain operating information from aplurality of autonomous vehicles, smart homes, and/or personalelectronic devices having the selected component or type of component.Such operating information may be limited to operating informationassociated with the selected component, or it may include additionaloperating information associated with the vehicles, smart homes, and/orpersonal electronic devices. The information may be obtained byaccessing a database 146 storing information regarding operation of aplurality of vehicles, smart homes, and/or personal electronic devicesin full or summary form. In some embodiments, the operating informationmay include operating data (including sensor data and/or control data),as described elsewhere herein. In other embodiments, the operatinginformation may include information derived from operating data and/orloss data associated with vehicle accidents and/or loss events,component failure, or other incidents involving the selected componentfor the plurality of vehicles, smart homes, and/or personal electronicdevices.

At block 806, the server 140 may identify occurrences of the selectedcomponent malfunctioning in the plurality of vehicles, smart homes,and/or personal electronic devices based upon the operating information.This may include identifying recorded occurrences of componentmalfunction events, occurrences of repairs associated with thecomponent, or occurrences of collisions or other loss events (which maybe further analyzed to determine whether the component wasmalfunctioning prior to the collision or other loss event). In someembodiments, this may include evaluating operating data associated withthe plurality of vehicles, smart homes, and/or personal electronicdevices to identify indications of component malfunctions based uponsignals that are out of an expected range or are inconsistent with othercontemporaneous signals associated with other components of the samevehicle, smart home, and/or personal electronic device.

At block 808, the server 140 may determine one or more risks ofcomponent malfunction based upon the identified occurrences ofmalfunctioning in the plurality of vehicles. The one or more risks maybe associated with probabilities of component malfunction occurrences,which may further be associated with types of component malfunctions(e.g., inaccurate operation, unresponsiveness, etc.). The one or morerisks may likewise be associated with locations of operation, times ofoperation, durations of the presence of the component in the autonomousvehicle, smart home, and/or personal electronic device, extent of use ofthe component in the autonomous vehicle, smart home, and/or personalelectronic device, or other relevant factors.

The duration of the presence of the component in the autonomous vehicle,smart home, and/or personal electronic device may be measured in totaltime, total operating time, or total distance traveled by the vehiclewith the component installed within the vehicle. The extent of use ofthe component in the autonomous vehicle, smart home, and/or personalelectronic device may be measured in total operating time or totaldistance traveled by the vehicle while the component was engaged inoperation of the vehicle.

At block 809, the server 140 may additionally or alternatively determineone or more risks of cyber-attack directed at the component of one ormore autonomous vehicles, smart homes, and/or personal electronicdevices. The server 140 may determine the software version and/oroperating system of the component. The database 146 may additionallyinclude an indication of a latest software version and/or operatingsystem version distributed by the component manufacturer, a date thelatest version was released, and/or a number of vulnerabilitiescorrected by the latest version. The corrected vulnerabilities may beorganized by severity (e.g., low, mid, high, critical, etc.). In somescenarios, several versions may have been released between the versionexecuting on the component and the latest version. In these scenarios,the controller may aggregate the vulnerabilities from each versionsubsequent to the currently executing version.

The risk of cyber-attack may be determined by generating a vulnerabilityscore indicating a risk level associated the known vulnerabilities inthe current software version and/or operating system version executingon the component. In an embodiment, one or more vulnerabilities may beassociated with particular functions and/or features that may bemaliciously controlled by exploiting the vulnerability. In thisembodiment, the vulnerability score may be further based upon theoperation and/or performance of the functions and/or features exposed inthe component's currently executing software version and/or operatingsystem version.

At block 810, the server 140 may further determine results associatedwith each identified occurrence of the component malfunctioning. Suchresults may include immediate and longer-term results, including bothevents (e.g., collisions) and non-events (e.g., no significant change inautonomous operation). Each result may be indicative of an impact of thecomponent malfunction on the operation of the vehicle.

Such impact may include an impact on a risk or severity of a vehiclecollision involving the vehicle and/or other loss event involving thevehicle, smart home, and/or personal electronic device. Such impact maysimilarly include an impact on a risk or severity of a loss of controlevent, on an inability to operate in a fully autonomous orsemi-autonomous mode, on a collision or loss of control event involvinganother vehicle, smart home, and/or personal electronic device in theoperating environment, or other aspects of autonomous control (e.g.,recognition and/or appropriate response to environmental conditions,pedestrians, other vehicles, etc.). The result may further includeinformation regarding the impact, such as occurrence and/or extent ofdamage to the vehicle, smart home, and/or personal electronic device,damage to other vehicles, smart homes, and/or personal electronicdevices, damage to other property, costs associated with repair ofdamage, injuries to passengers of the vehicle or other vehicles,injuries to pedestrians and/or passersby, or costs associated withinjuries.

In some embodiments, determining the results associated with theidentified occurrences of component malfunctioning may includedetermining the influence of mitigation by one or more actions of thevehicle, smart home, and/or personal electronic device to offset thecomponent malfunction. Such mitigating actions taken by the vehicle,smart home, and/or personal electronic device in response to thecomponent malfunction may include making adjustments to the operation ofone or more autonomous operation features associated with themalfunctioning component, placing restrictions or limitations on use ofthe one or more autonomous operation features, or engaging additionalcomponents to compensate for the malfunction. Where the malfunctioningcomponent is or includes a sensor, the mitigating actions may includeignoring sensor data from such sensor and/or using sensor data from oneor more redundant sensors (which may be of the same general type as themalfunctioning sensor) to operate the vehicle, smart home, and/orpersonal electronic device. In some embodiments, redundant sensors orother components may not be activated until the on-board computer 114,smart home controller 185, and/or personal electronic device 189determines that the component is malfunctioning.

In some embodiments, the mitigating actions may be associated with aversion of a software program associated with an autonomous operationfeature or a version of an operating system for autonomous operation ofthe vehicle, smart home, and/or personal electronic device. For example,newer versions of software used by the vehicle, smart home, and/orpersonal electronic device may include additional functionality to takemitigating actions not included in older versions of such software. Asanother example, new version of software may reduce the vulnerability ofthe vehicle, smart home, and/or personal electronic device tocyber-attacks.

At block 812, the server 140 may determine and/or generate a componentprofile based upon the determined risks of component malfunction andresults of component malfunction. The component profile may indicate oneor more combinations of risk levels, including cyber-attack risk levels,and impacts associated with malfunctions of the component, which maydepend upon one or more settings associated with the component. Suchcombinations may be further associated with a plurality of operatingconditions, as well as other aspects of the vehicle, smart home, and/orpersonal electronic device. Such conditions and aspects may havesignificant influence on the probability and severity of incidentsresulting from component malfunctions. Operating conditions of vehicleuse, smart home use, and/or personal electronic device use may includeoperating environments through which a vehicle travels, an environmentproximate to the smart home, and/or environments in which the personmonitored by the personal electronic device travels, which may includelocation, weather, traffic, road type, time of day, etc.

Aspects of the vehicle, smart home, and/or personal electronic devicemay include fixed or adjustable characteristics of a vehicle, smarthome, and/or personal electronic device that may interact with thecomponent, including other components of the vehicle, smart home, and/orpersonal electronic device, extent of use of autonomous operationfeatures, settings of the autonomous operation features used, etc. Forexample, each combination in the component profile may be associatedwith a configuration of additional components (which may includesettings thereof) that interact with the component to operate thevehicle, smart home, and/or personal electronic device.

In some embodiments, the component profile may include one or morescores associated with risks and results associate with the componentunder one or more sets of conditions and/or aspects. Such scores may beindicative of an expected value of the impact of component malfunctions,including cyber-attacks. In further embodiments, the component profilemay additionally, or alternatively, indicate an expected usable lifetimeof the component. Such expected usable lifetime may be associated with aduration of time or extent of distance traveled by the vehicle with thecomponent installed or functioning before the component reaches apredetermined failure rate (e.g., 50% probability of malfunction, 70%probability of malfunction, etc.).

At block 814, the server 140 may determine whether there is a furthercomponent of the vehicle 108, the smart home 187, and/or the personalelectronic device 189 to assess. When a further component is identified,the method 800 may continue with selection of the next component toassess (block 802). When no further component is identified, the method800 may terminate or may proceed to determination and/or generation of arisk profile for the autonomous vehicle, smart home, and/or personalelectronic device (block 816).

At block 816, in some embodiments, the server 140 may determine and/orgenerate a risk profile for the vehicle 108, the smart home 187, and/orthe personal electronic device 189 based upon a plurality of componentprofiles determined for a plurality of autonomous operation componentsof the vehicle 108, the smart home 187, and/or the personal electronicdevice 189. The risk profile may be determined based upon the entries inthe plurality of component profiles. The risk profile may be generatedby the server 140 by appending the component profiles, or the riskprofile may be generated by the server 140 by processing combining theentries in the component profiles. Combining the entries in thecomponent profiles may include generating conditional risk levels orimpact estimates, as well as conditional expected value estimatesassociated with various conditions and/or components. In someembodiments, some or all of the entries in the risk profile mayrepresent total risk levels or total expected values that incorporaterisks and results for a plurality of components of the vehicle 108, thesmart home 187, and/or the personal electronic device 189.

At block 818, in some embodiments, the server 140 may determine one ormore actions to reduce one or more risks associated with autonomousoperation based upon the determined risk profile for the vehicle 108,the smart home 187, and/or the personal electronic device 189. The oneor more actions may be associated with reducing the risks (or results)associated with malfunctions of one or a plurality of the components ofthe vehicle 108, the smart home 187, and/or the personal electronicdevice 189. To determine actions to reduce risks the server 140 maydetermine one or more repairs, upgrades, replacements, or updates thatmay be made to the one or more components. For example, the server 140may identify a software version associated with an autonomous operationfeature that would reduce the expected damage caused by a componentfailure and/or reduce the risk of exposure of the component to acyber-attack.

Based upon such determination of actions to reduce risks or impacts fromcomponent malfunctions and/or cyber-attacks, the server 140 may furtherdetermine additional information necessary to implement such actions.For example, information regarding repair or replacement options orcosts may be determined, as may information regarding repair serviceproviders. Where the one or more actions include updating or upgradingsoftware, the server 140 may determine to automatically update orupgrade the software when the vehicle 108, the smart home 187, and/orthe personal electronic device 189 is not in operation, and/or at a timegenerally associated with low risk of impacting other components (e.g.,around 3 A.M.). The one or more actions may thus include causing thevehicle 108, the smart home 187, and/or the personal electronic device189 to update or upgrade one or more components automatically.Similarly, the one or more actions may include scheduling an appointmentto repair or replace one or more components of the vehicle 108, thesmart home 187, and/or the personal electronic device 189. The one ormore actions may further include causing the vehicle 108 to travel in afully autonomous mode to a repair service provider for repair orreplacement of the one or more components.

Alternatively, or additionally, the one or more responses may includegenerating one or more messages regarding the components forpresentation to an owner or operator of the vehicle 108, an occupant orowner of the smart home 187, and/or the person monitored by the personalelectronic device 189. Such messages may include notifications orrecommendations regarding the determined repairs, upgrades,replacements, or updates that may be made to the one or more components.The messages may similarly include recommendations regarding usage ofthe one or more components, such as recommendations of conditions and/orsettings for use of such components to reduce risk. Such messages mayfurther include information regarding costs associated with therecommendations, which may include cost savings or reductions in costsassociated with a vehicle insurance policy.

At block 820, in some embodiments, the server 140 may implement thedetermined one or more actions. This may include generating andcommunicating messages to owners and/or operators of the vehicle 108,the occupants and/or owners of the smart home 187, and/or the personsmonitored by the personal electronic device 189 for presentation via adisplay 202. In some embodiments, such actions may be implemented byscheduling appointments to repair or replace components, as well ascontrolling vehicles to travel to service locations for suchappointments. In further embodiments, the actions may be implemented byautomatically updating or upgrading software associated with one or moreautonomous operation features of the vehicle 108, the smart home 187,and/or the personal electronic device 189. The server 140 maycommunicate with the on-board computer 114, the smart home controller185, and/or the personal electronic device 189 to implement suchactions.

Exemplary Component Repair Methods

FIG. 9 illustrates a flow diagram of an exemplary autonomous componentrepair method 900 for identifying and repairing malfunctioningcomponents of an autonomous vehicle 108, a smart home 187 and/or apersonal electronic device 189. Such method 900 may be utilized toautomatically detect and correct malfunctions affecting components of avehicle 108, a smart home 187 and/or a personal electronic device 189that are associated with autonomous operation. Unlike traditionaldevices, autonomous vehicles, smart homes and/or personal electronicdevices may be capable of self-diagnosis using the on-board computer 114and/or the smart home controller 185. In embodiments associated withautonomous vehicles, the autonomous vehicle 108 may be further capableof automatic travel to facilities capable of repairing or replacingmalfunctioning components. In circumstances in which malfunctions arenot so severe as to preclude autonomous operation, the on-board computer114 may automatically schedule an appointment for such repairs andcontrol the vehicle 108 to travel to such appointment, either with orwithout the involvement of a vehicle owner or operator. When themalfunctions preclude autonomous operation, or in embodiments associatedwith the smart home 187 and/or the personal electronic device 189,however, the on-board computer 114 may nonetheless alert an owner,occupant, or operator to the situation, which may also includescheduling an appointment for the vehicle 108, the smart home 187 and/orthe personal electronic device 189 to be repaired. Although the method900 is described below as being performed by the on-board computer 114,the smart home controller 185, the personal electronic device 189 forsimplicity, it should be understood that one or more mobile devices 110or servers 140 may alternatively, or additionally, be used to performpart or all of the process.

The exemplary autonomous component repair method 900 may begin byreceiving data associated with operation of the vehicle 108, the smarthome 187 and/or the personal electronic device 189 (block 902). When anindication of an unusual condition is identified in the received data, adetermination to evaluate one or more autonomous operation components ofthe vehicle 108, the smart home 187 and/or the personal electronicdevice 189 may be made (block 904), and status data regarding the one ormore components may be obtained (block 906). A malfunction of at leastone of the components may be determined based upon the status data(block 908). One or more repairs may then be determined to correct themalfunction (block 910), and such repairs may be scheduled with a repairservice provider (block 912). In embodiments associated with theautonomous vehicle 108, if the vehicle 108 is able to operate fullyautonomously despite the malfunction (block 914), the vehicle 108 maytravel autonomously to a repair location associated with the scheduledappointment (block 916). If the vehicle 108 is unable to operate fullyautonomously with the malfunction (block 914), an alert regarding themalfunction may be generated and presented to a vehicle owner oroperator (block 918). In some embodiments, the repair may be scheduledor rescheduled for a current location of the vehicle 108 when thevehicle 108 is unable to operate fully autonomously (block 920). Infurther embodiments, insurance or other coverage for the repair may bedetermined and payments made automatically (block 922).

At block 902, the on-board computer 114, the smart home controller 185and/or the personal electronic device 189 may receive data associatedwith operation of the vehicle 108, the smart home 187 and/or thepersonal electronic device 189. This may include receiving operatingdata associated with the vehicle 108, the smart home 187 and/or thepersonal electronic device 189, or other information regardingoperation. In some embodiments, the data may be received from a vehicleoperator and/or smart home occupant. A vehicle operator and/or smarthome occupant may manually enter input or select an option indicative ofa general operating status of the vehicle 108, the smart home 187 and/orthe personal electronic device 189. For example, a vehicle operatorand/or smart home occupant may select an option to begin a diagnosticand repair routine via the on-board computer 114, the smart homecontroller 185, the personal electronic device 189, and/or a mobiledevice 110, such as when the vehicle operator and/or smart home occupantobserves something to be amiss in autonomous operation. In furtherembodiments, the data may include a summary indicator of whether anunusual condition or an incident has occurred, which summary indicatormay be generated by a monitoring or response method, as describedelsewhere herein (particularly with reference to method 500).

At block 904, the on-board computer 114, the smart home controller 185,and/or the personal electronic device 189 may determine to assess one ormore autonomous operation components of the vehicle 108, the smart home187 and/or the personal electronic device 189 based upon the receiveddata. In some embodiments, the determination may be based upon anindication in the data of an anomaly associated with part of thereceived data related to the one or more components. For example, adiscrepancy between sensor data received from two or more sensors may bedetected, indicating that at least one of the two or more sensors may bemalfunctioning without directly indicating which sensor or sensors aremalfunctioning. In further embodiments, determining to assess the one ormore autonomous operation features may include determining theoccurrence of a loss-event, such as a collision involving the vehicle108, a loss of control of the vehicle 108, a near collision of thevehicle 108 with an object within a threshold distance, damage to thevehicle 108, the smart home 187 and/or the personal electronic device189, and/or the presence of unsafe conditions associated with the smarthome 187. The determination may further include determining whichcomponents to assess based upon the received data, such as bydetermining a collision involving a front portion of the vehicle 108 orloss occurred in the basement of the smart home 187.

At block 906, the on-board computer 114, the smart home controller 185and/or the personal electronic device 189 may obtain status dataregarding the one or more components of the vehicle 108, the smart home187 and/or the personal electronic device 189. This may includerequesting, accessing, or receiving such status data from a programmemory 208 or from the component itself. The status data may includeoperating data associated with the one or more components, such assensor data from a plurality of sensors 120 of the vehicle 108, thesmart home 187 and/or the personal electronic device 189. In someembodiments, the status data may be indicative of a self-diagnosis ofthe component or may be received over a sample period of time.

In further embodiments, the status data may be received in summary form,such as a summary of operating data associated with the one or morecomponents (e.g., summaries of control decisions made by an autonomousoperation feature). Such status data may further include or beassociated with information regarding conditions in the autonomousoperation environment. In embodiments in which a loss-event isdetermined to have occurred, the on-board computer 114, the smart homecontroller 185 and/or the personal electronic device 189 may request thestatus data in response to such determination of the incident.

At block 908, the on-board computer 114, the smart home 187 and/or thepersonal electronic device 189 may evaluate the one or more componentsusing the status data to determine at least one of the components ismalfunctioning. This determination may include identifying at least oneof the components that is malfunctioning by iteratively evaluating theone or more components against predetermined or dynamically determinedexpected ranges of operation, such as discussed elsewhere herein(particularly with reference to method 700). This determination mayfurther include identifying a type of malfunction, cause of themalfunction, and/or extent of the malfunction. In some embodiments, thedetermination may include a prediction of future failure of thecomponent based upon current or prior status data regarding thecomponent. Such prediction may be further based upon a comparison withinformation regarding similar components of a plurality of othervehicles, smart homes, or personal electronic devices, which may bereceived from the server 140 or the database 146. In some embodiments,the server 140 may determine such prediction and communicate theprediction of future failure of the component.

At block 910, the on-board computer 114, the smart home controller 185and/or the personal electronic device 189 may determine one or morerepairs to correct the determined malfunction. Such repairs may includeadjusting or replacing a component or portion thereof. The repairs maybe determined based upon the status data or information derivedtherefrom, such as the cause of the malfunction. The repairs may furtherbe based upon information regarding repairs made to a plurality of othervehicles, smart homes, and/or personal electronic devices having similarcomponent malfunctions. In some embodiments, determining the one or morerepairs may include further determining one or more requirementsassociated with the repairs that indicate parts required for the repairsor skill levels required for the repairs.

In some embodiments, the one or more repairs may be determined by theserver 140 based upon information received from the on-board computer114, the smart home 187 and/or the personal electronic device 189. Infurther embodiments, the determined repairs may be verified by a humanreviewer, such as an agent of an insurer. Such verification may beperformed remotely or at a repair facility or other site associated withthe reviewer, in which case the vehicle 108 may be caused to travel tothe site for review. In yet further embodiments, external photographicor video evidence of damage to the vehicle 108, the smart home 187and/or the personal electronic device 189 may be captured and stored bya human reviewer, regardless of whether the human reviewer is requiredto verify the determined repairs. This may be particularly useful wherea third party may be liable for the malfunction.

At block 912, the on-board computer 114, the smart home controller 185and/or the personal electronic device 189 may automatically schedule anappointment with a repair service provider to perform the one or morerepairs. Such scheduling may be performed in conjunction with the server140, which may be instructed by the on-board computer 114, the smarthome controller 185, and/or the personal electronic device 189 toschedule an appointment based upon the determined one or more componentmalfunctions and/or repairs. Scheduling the appointment may includefirst identifying one or more repair service providers capable ofperforming the determined one or more repairs to correct the componentmalfunction. Identifying the repair service providers may includedetermining that the repair service providers are capable of meeting oneor more requirements associated with the repairs that indicate partsrequired for the repairs or skill levels required for the repairs. Insome embodiments, the on-board computer 114, the smart home controller185, the personal electronic device 189 and/or server 140 may provideinformation regarding the one or more determined component malfunctionand repairs to the repair service provider in order to schedule theappointment.

If multiple repair service providers are determined to be capable ofperforming the repair, one of the repair service providers may beselected based upon scheduling availability, cost, location, qualitymetrics associated with previous repairs, or other criteria. Theappointment may be scheduled for a service time and service location forthe repairs to be performed immediately or at some future pointfollowing scheduling. In some embodiments, the appointment may bescheduled for a time when the vehicle 108, the smart home 187 and/or thepersonal electronic device 189 is ordinarily not in use in order tominimize the impact of the repairs on owners, occupants, or operators.

In further embodiments, the appointment may be presented to an owner,occupant, or operator for confirmation prior to being set. In yetfurther embodiments, alternative transportation may be scheduled for thevehicle owner or operator during the scheduled repairs. Such alternativetransportation may include taxi service, temporary vehicle-sharingmembership, vehicle rental, or similar temporary replacementtransportation.

At block 914, in embodiments associated with the vehicle 108, theon-board computer 114 may determine whether the vehicle 108 is capableof fully autonomous operation to travel to the repair service providerlocation. In some embodiments, such determination may be made prior toor while scheduling the appointment, which may influence the selectionof the time and location of the appointment. The determination ofwhether the vehicle 108 is capable of fully autonomous operation mayinclude determining that the risk associated with fully autonomousoperation of the vehicle 108 by its autonomous operation features isbelow a risk level below a risk threshold associated with safeoperation, despite the component malfunction. When the vehicle 108 isdetermined to be capable of fully autonomous operation, the on-boardcomputer 114 may control the vehicle to travel to the service locationfor the scheduled appointment (block 916). If fully autonomous operationof the vehicle 108 by its autonomous operation features would result ina risk above the risk threshold, the vehicle 108 may be determined unfitfor fully autonomous operation. When the vehicle 108 is determined notto be capable of fully autonomous operation, the on-board computer 114may alter an owner or operator of the vehicle 108 of the componentmalfunction and inability of the vehicle 108 to operate fullyautonomously (block 918).

At block 916, in embodiments associated with the vehicle 108, theon-board computer 114 may control the vehicle 108 to travel fullyautonomously to the service location to arrive at or before thescheduled service time using the autonomous operation features. Theon-board computer 114 may select autonomous operation features orsettings associated therewith to minimize risks or risk-weighted impactsassociated with accidents during autonomous operation to the scheduledappointment. This may include limiting or eliminating use of one or moreautonomous operation features that depend upon the one or moremalfunctioning components.

The on-board computer 114 may further select a route to minimize risksof damage or injury, such as by avoiding highways or road segmentspreviously determined to be associated with high risk for autonomousoperation. The on-board computer 114 may further adjust settings toreduce risk, such as by limiting vehicle operation to travel below aspecific speed or only travelling during daylight hours withoutprecipitation. To meet such requirements, in some embodiments, theon-board computer 114 may cause the vehicle 108 to travel to the servicelocation in advance of the service time. Information regarding thescheduled appointment or the fully autonomous operation of the vehicle108 may be presented to an owner or operator of the vehicle 108 forreview, adjustment, or approval.

At block 918, the on-board computer 114, the smart home 187 and/or thepersonal electronic device 189 may generate an alert to an owner,occupant, or operator of the vehicle 108, the smart home 187 and/or thepersonal electronic device 189 when the vehicle 108, the smart home 187and/or the personal electronic device 189 cannot operate fullyautonomously to travel to the service location. The on-board computer114, the smart home 187 and/or the personal electronic device 189 mayfurther cause the alert to be presented to the owner, occupant, oroperator by a display of the vehicle 108, the smart home controller 185,the personal electronic device 189, mobile device 110, or othercomputing device associated with the owner, occupant, or operator. Thealert may include information regarding the one or more malfunctionsassociated with the components of the vehicle 108, the smart home 187and/or the personal electronic device 189, such as a summary ofmalfunctions or causes of the malfunctions. The alert may furtherinclude one or more repair recommendations. In some embodiments,information regarding the recommended repairs may be included in thealert, such as typical costs, time, or parts associated with suchrepairs. In further embodiments, information regarding one or morerepair service providers may be provided, and one or more proposedappointments may be recommended. In yet further embodiments, the alertmay include a request to the owner, occupant, or operator to confirm,reschedule, or cancel an automatically scheduled appointment with arepair service provider.

At block 920, in some embodiments, the on-board computer 114, the smarthome 187 and/or the personal electronic device 189 may schedule anappointment to repair the one or more malfunctioning components at aservice location not associated with the repair service provider. Suchservice location may include a current location of the vehicle 108, thesmart home 187 and/or the personal electronic device 189 or a usualparking or garaging location of the vehicle 108. Thus, the vehicle 108,the smart home 187 and/or the personal electronic device 189 may berepaired without traveling to the repair service provider, which may beof particular value when the vehicle 108 is incapable of fullyautonomous operation because of the one or more malfunctions. This mayinclude automatically rescheduling the appointment with the repairservice provider, or this may include changing the service locationbefore scheduling the appointment (or before finalizing or confirmingthe appointment).

At block 922, in some embodiments, the on-board computer 114, the smarthome 187 and/or the personal electronic device 189 or server 140 maydetermine an insurance policy coverage or other coverage for the one ormore repairs. Such coverage may be determined based at least in partupon a determined cause of each malfunction. For example, a coverage forweather-related damage may be determined to apply to the one or morerepairs when the one or more malfunctions are determined to have beencaused by freezing temperatures or hail damage. In further embodiments,a payment may be automatically made to the repair service provider forthe repair work based upon the determined coverage. This may befacilitated by one or more servers 140, which may cause a transfer offunds to be made to an account associated with the repair serviceprovider. Such transfers may be made following completion of therepairs. The one or more server 140 may further cause a deduction to bemade from an account associated with the vehicle 108, the smart home 187and/or the personal electronic device 189 for a copayment or deductiblepayment associated with the coverage in yet further embodiments.

Exemplary Malfunction Detection Methods

FIG. 10 illustrates a flow diagram of an exemplary autonomousenvironment control software evaluation method 1000 for testing thequality of control decisions generated by autonomous operation featureof an autonomous vehicle 108, a smart home 187, and/or a personalelectronic device 189. Such method 1000 may be implemented to assess theoperation of software components used in autonomous operation outside ofa vehicle, a smart home, and/or a personal electronic device. Suchtesting may allow new software or software version updates to beevaluated in a realistic computing environment without the hazardsassociated with testing in a vehicle, a smart home, and/or a personalelectronic device. In some embodiments, the test environment may includeemulation of an autonomous operating system, such as the on-boardcomputer architecture 114 and/or the smart home controller 185,operating at an artificially accelerated system clock speed tofacilitate faster testing of control software under a variety ofconditions.

Although the method 1000 is described below as being performed by oneserver 140 for simplicity, it should be understood that one or moremobile devices 110, on-board computers 114, smart home controllers 185,personal electronic device 189, or servers 140 may alternatively, oradditionally, be used to perform part or all of the process. Forexample, multiple test scenarios representing a plurality of testconditions may be simultaneously run using a plurality of servers 140(or a plurality of processors 162) to reduce the total time required toevaluate the software features.

The exemplary method 1000 may begin with selection of one or moreautonomous operation features to test (block 1002). Based upon theselection of autonomous operation features, computer instructionsassociated with such autonomous operation features may be retrieved froma memory storage (block 1004). The computer instructions may include oneor more software routines associated with the selected autonomousoperation features. A further selection of test conditions to be used intesting the autonomous operation features may be received (block 1006),which may be associated with types of vehicles, smart homes, and/orpersonal electronic devices, other software features, or environmentalconditions of a virtual operating environment mimicking a vehicleoperating environment. Simulated input data may then be generated basedupon the selected autonomous operation features and test conditions(block 1008). An emulation of an autonomous environment operating systemmay be started to test the software routines (block 1010), and thesoftware routines may be implemented within the emulated operatingsystem (block 1012).

Once the software environment is prepared, the test may be performed bypresenting the simulated input data to the software routines (block1014), processing the simulated input data by the software routines(block 1016), and recording output data received from the softwareroutines (block 1018). Such output data may include control commandsgenerated by the software routines configured to directly or indirectlycontrol autonomous operation when executed within an autonomousenvironment during autonomous operation using the associated autonomousoperation feature. One or more quality metrics indicative ofeffectiveness of the evaluated autonomous operation features may becalculated for the software routines based upon the recorded output data(block 1024). In some embodiments, baseline output values may beretrieved (block 1022) and used in generating and/or determining the oneor more quality metrics by comparison with the recorded output data(block 1024).

At block 1002, a user may select one or more autonomous operationfeatures to evaluate. The selection may be made from a list ofautonomous operation features or groups of autonomous operationfeatures. In some embodiments, the user may select the features via auser interface of a display 202, which selection may be communicated tothe server 140. In further embodiments, the user may implicitly selectone or more features to evaluate by storing or loading the features in amemory of the server 140 for testing. This may include storingcomputer-readable instructions associated with the autonomous operationfeatures in a directory location that may be accessed for testing. Inyet further embodiments, the user may select a directory location toselect the computer-readable instructions stored at such location.

At block 1004, the server 140 may access computer-readable instructionsassociated with the selected one or more autonomous operation featuresin response to receiving the user selection. Such computer-readableinstructions may comprise one or more software routines that may beimplemented within an autonomous environment operating system, such asone running on an on-board computer 114 of an autonomous vehicle 108and/or one running on a smart home controller 185 of a smart home 187,to perform control, monitoring, assessment, communication, and/orsimilar operations in support of autonomous operation. Although referredto as software routines for simplicity, such computer-readableinstructions may include subroutines, programs, applications, scripts,or similar executable or callable code segments. Accessing the softwareroutines may include retrieving the computer-readable instructionsstored in one or more directory locations. In some embodiments, thesoftware routines may be accessed from a remote storage device, such asanother server connected to the server 140 via the network 130.

At block 1006, the server 140 may receive a selection of test conditionsfor evaluating the one or more autonomous operation features. The usermay select the test conditions by selecting one or more indicatorsrepresenting the test conditions, or the server 140 may automaticallyselect the test conditions based upon a configuration file or otherstored information. The test conditions may indicate parameters, scope,or duration of testing of the autonomous operation features. In someembodiments, the selection of test conditions may include selection of amake and/or model of an autonomous vehicle, an on-board computer, asmart home controller, or an autonomous environment operating system (orversion thereof). This may include information regarding sensors orsensor data available for use by the autonomous operation features(e.g., number and type of sensors, operating status of sensors, orconfiguration of sensors within a simulated environment).

In further embodiments, the selection of test conditions may includeindicating one or more types of environmental conditions to mimic in thesimulated input data to be presented to the software routines duringtesting. Such environmental conditions may include conditions relatingto time of day (e.g., daylight levels, glare, or headlights of othervehicles), weather (e.g., wind, precipitation, visibility, temperature,or other weather conditions), road type (e.g., highway, residential,urban, or rural), road integrity (e.g., road material, ice, poolingwater, or potholes), traffic (e.g., congestion, mean or median speed, oraverage vehicle spacing), neighborhood (e.g., urban, suburban, apartmentcomplex, or agricultural) construction (e.g., lane closures, temporarytraffic patterns, or active construction work), and/or other similarconditions that affect effectiveness of autonomous operation features incontrolling a vehicle and/or smart home. The environmental conditionsmay be specified at various levels of detail or in groupings, such asurban rush hour conditions or rural winter storm conditions.

At block 1008, the server 140 may generate simulated input data basedupon the selected test conditions. The simulated input data may begenerated as one or more sets of test data associated with testconditions. The test data may include simulated sensor data, such as aplurality of simulated sensor data points to be presented to thesoftware routines as inputs during testing. The test data may furtherinclude simulated control data from one or more other autonomousoperation features that may be provided to the software routines asinputs during testing, as well as simulated communication data from oneor more communication components. In some embodiments, the set of testdata may include a plurality of subsets of test data, each such subsetrepresenting a combination of conditions (e.g., daylight urban drivingin moderate congestion during clear weather, daylight urban driving inmoderate congestion during rain, daylight urban household during snow,etc.). In this manner, the selected autonomous operation features may betested in a range of environments in an efficient manner, withoutrequiring the user to specify each potential environmental condition.Similarly, the subsets of test data may be used to present a variety ofscenarios with the same combination of conditions. For example, normaloperation may be tested by one subset of test data, while response tothe simulated vehicle being cut off by another vehicle within thesimulation may be tested using another subset of test data.

The test data may include one or more sequences of simulated datasignals, such as sensor data signals. Each sequence of simulated datasignals may simulate a time series of continuous or discrete data over atime period during vehicle operation. For example, a sequence may beassociated with a time series of data points over a time intervalrepresenting sensor readings from an accelerometer within the vehicle.Another sequence may be associated with a time series of data pointsover the same time interval of a proximity sensor at a location withinthe vehicle and/or smart home. The sequences may be standardizedsequences generated in advance and stored in a database 146, to beretrieved by the server 140 to generate the set of test data. Suchretrieved sequences may include recorded sequences of sensor data fromactual autonomous operation that has previously occurred and beenrecorded by on-board computers 114 of a plurality of vehicles 108 and/ora plurality of smart home controller 185 from a plurality of smart homes187. Such recorded sensor data may be associated with ordinaryautonomous operation or incidents involving the vehicles and/or smarthomes (e.g., loss of control situations, collisions, loss-events, etc.).

In some embodiments, the sequences may be generated by the server 140based upon the received selections of autonomous operation features andtest conditions, such as by generating expected sensor data based upon asimulation of an autonomous operation in a virtual environment meetingthe criteria indicated by the selected test conditions. The set of testdata may be generated by combining a plurality of sequences of simulateddata signals, which sequences may be matched in time to be concurrentwithin the simulated test environment. In some embodiments, thecombination of sequences may include a simple aggregation of such signalsequences. In further embodiments, combining the sequences may includegenerating one or more summary sequences or signals from two or more ofthe retrieved sequences, or combining the sequences may includemodifying one or more of the sequences based upon other retrievedsequences.

At block 1010, the server 140 may start an emulator program to mimic anautonomous environment operating system. The emulator may be configuredto mimic a specific version of an autonomous environment operatingsystem running on a particular make and model of a smart home controller185 and/or an on-board computer 114, which may be associated with aparticular make, model, and year of an autonomous vehicle. The emulatormay perform operations to execute the computer-readable instructions ofthe software routines on the hardware architecture of the server 140.This may include translating signals from the sets of test data into aformat usable by the software routines and translating the output of thesoftware routines for storage in the program memory 160 or database 146.Such translation may include interpolation or sampling of data points.In some embodiments, the emulator program may be configured to operateat an accelerated speed in order to process the test data faster thanreal-time. This may include keeping a separate internal system clockwithin the emulator program that is used by the software routines. Suchinternal system clock may progress faster than an external clock of theserver 140. By operating at an accelerated speed, the emulator programmay enable the software routines to process the simulated input data inless time than the time interval nominally associated with the data.

At block 1012, the emulator may implement the one or more softwareroutines associated with the selected autonomous operation features.This may include accessing a directory location where thecomputer-readable instructions specifying the software routines arestored to load the software routines in the emulated environment. Thesoftware routines may be executed within the emulator to process thesimulated input data in a manner similar to operation within an on-boardcomputer 114 of an autonomous vehicle 108 and/or a smart home controller185 of a smart home 187.

At block 1014, the emulator may present the simulated input data asinputs to the one or more software routines within the emulatedenvironment. In some embodiments, the emulator may receive the set oftest data from another program or routine running on the server 140. Asnoted above, the simulated input data may include simulated orprerecorded values of sensor data, control data, communication data, ora combination of such data. The emulator may parse the set of test datato separate the types of simulated input data for separate presentationto the software routines. For example, the sensor data may include datasequences with different periodicity from the update period of controldata signals with in the data set. The emulator may then align thepresentation of such various simulated input data to the softwareroutines to represent a coherent set of inputs.

In some embodiments, the emulator may further implement one or moreadditional software routines to generate control data as further inputsto the software routines. Such additional software routines may bepresented with part or all of the simulated input data as inputs, andthe outputs generated by such additional software routines may bepresented as inputs to the one or more software routines.

At block 1016, the one or more software routines may process thereceived inputs to generate output data within the emulator environment.The output data generated by the software routines may include outputsignals indicative of conditions determined by the software routines,control signals configured for controlling autonomous components, and/orother data generated by the software routines as inputs into furthersoftware or hardware components configured for controlling autonomouscomponents. The one or more software routines may run within theemulator program on the server 140 to produce such output data in thesame manner that the software routines would run within the autonomousenvironment operating system, such as on an on-board computer 114 duringvehicle operation and/or a smart home controller 185 during smart homeoperation. In some embodiments, the emulator program may control one ormore settings of the autonomous operation features associated with thesoftware routines by adjusting parameter variable values stored withinthe emulated environment and accessed by the software routines duringdata processing.

At block 1018, the emulator may cause the output data or indicationsassociated with the output data to be stored by the server 140. This mayinclude sampling or translating the output data from the softwareroutines to prepare the output data for storage by the server 140. Theserver 140 may receive the prepared output data or indicationsassociated therewith and store such received data in the program memory160 or database 146 for further analysis. The output data may betime-stamped or otherwise associated with sequences of simulated datasignals in the set of test data. In some embodiments, the indicationsassociated with the output data may include indications of errors orfailures of the software routines. For example, the indications couldincluding information regarding stack overflow, infinite looping,out-of-range values, or other events associated with failures of thesoftware routines to operate properly or perform vehicle controlfunctions in an effective manner.

In some embodiments, at block 1020, the emulator (or other applicationrunning on the server 140) may further use the output data generated bythe one or more software routines in determining or adjusting thesimulated input data to present to the software routines as inputs. Thismay include providing previously generated output data as input data(e.g., prior period control data generated by a software routine may beused as an input for generating current period control data). This maylikewise include adjusting simulated sensor data or control data fromadditional software routines based upon output data generated by thesoftware routines. For example, the server 140 may run a virtual vehicleenvironment simulation application that interfaces with the emulator toprovide simulated input data to the emulator and to model movement of avirtual vehicle within a virtual environment based upon the output datagenerated within the emulator. Thus, output generated by the softwareroutines may be used to control a virtual position of the virtualvehicle relative to other objects within the virtual environment, whichupdated position may then be used to generate further simulated inputdata. Similarly, the server 140 may run a virtual smart home environmentsimulation application that interfaces with the emulator to providesimulated input data to the emulator to model movement of objects withinthe virtual environment proximate to or within a smart home. Forexample, the virtual smart home environment simulation application maysimulate and/or model a virtual fire on a neighboring property. Theadjusted or updated simulated input data may then be presented to theone or more software routines running within the emulator program (block1014).

In further embodiments, at block 1022, the server 140 may accessbaseline output values associated with the selected test conditions fromthe database 146. The baseline output values may be indicative ofordinary or acceptable functioning of the selected one or moreautonomous operation features under the selected conditions. Thebaseline output values may be determined based upon calculated outputvalues required for safe autonomous operation of a vehicle, smart home,and/or personal electronic device, or the baseline output values may bedetermined from analysis of data indicating actual output values fromactual operation of a plurality of autonomous vehicles, smart homes,and/or personal electronic devices. In some embodiments, the baselineoutput values may be associated with test or actual output values fromanother version of a related software routine associated with anautonomous operation feature. For example, output values for a currentversion of control software routines associated with an autonomousoperation feature that had been determined by previous testing (i.e.,using the method described herein) may be used as baseline output valuesfor comparison against output data generated by testing a new version orupdate to the control software routines associated with the sameautonomous operation feature.

At block 1024, the server 140 may generate and/or determine one or morequality metrics indicative of effectiveness of the evaluated autonomousoperation features in controlling an autonomous vehicle. The qualitymetrics may be generated and/or determined based upon the output datarecorded during evaluation of the software routines associated with theautonomous operation features. In some embodiments, such quality metricsmay be indicative of risks associated with the autonomous operationfeatures. The quality metrics may be generated and/or determined bycomparison of the recorded output data with output values indicative ofeffective control of an autonomous vehicle, smart home, and/or personalelectronic device by the autonomous operation features. As discussedabove, some embodiments may generate and/or determine the qualitymetrics by comparing the recorded output data with baseline outputvalues.

Such comparison may include determining one or more measures ofdifferences between record output and baseline output values, which maythen be used to determine the one or more quality metrics. For example,the quality metric may be determined as a measure of magnitude of thedifferences between the record output and baseline output values. Thedifferences may be indicative of improvement or deterioration of theoperation of the tested software routines relative to the baselineperformance indicated by the baseline output values. The determinedquality metrics may be stored or presented to the user for review. Oncethe quality metrics have been generated, the method 1000 may terminate.

Exemplary Methods of Determining Risk Using Telematics Data

As described herein, telematics data may be collected and used inmonitoring, controlling, evaluating, and assessing risks associated withautonomous or semi-autonomous operation of a vehicle 108. In someembodiments, the Data Application installed on the mobile computingdevice 110 and/or on-board computer 114 may be used to collect andtransmit data regarding vehicle operation. This data may includeoperating data regarding operation of the vehicle 108, autonomousoperation feature settings or configurations, sensor data (includinglocation data), data regarding the type or condition of the sensors 120,telematics data regarding vehicle regarding operation of the vehicle108, environmental data regarding the environment in which the vehicle108 is operating (e.g., weather, road, traffic, construction, or otherconditions). Such data may be transmitted from the vehicle 108 or themobile computing device 110 via radio links 183 (and/or via the network130) to the server 140. The server 140 may receive the data directly orindirectly (i.e., via a wired or wireless link 183 e to the network 130)from one or more vehicles 182 or mobile computing devices 184. Uponreceiving the data, the server 140 may process the data to determine oneor more risk levels associated with the vehicle 108.

In some embodiments, a plurality of risk levels associated withoperation of the vehicle 108 may be determined based upon the receiveddata, using methods similar to those discussed elsewhere herein, and atotal risk level associated with the vehicle 108 may be determined basedupon the plurality of risk levels. In other embodiments, the server 140may directly determine a total risk level based upon the received data.Such risk levels may be used for vehicle navigation, vehicle control,control hand-offs between the vehicle and driver, settings adjustments,driver alerts, accident avoidance, insurance policy generation oradjustment, and/or other processes as described elsewhere herein.

In some aspects, computer-implemented methods for monitoring the use ofa vehicle 108 having one or more autonomous operation features and/oradjusting an insurance policy associated with the vehicle 108 may beprovided. In some embodiments, the mobile computing device 110 and/oron-board computer 114 may have a Data Application installed thereon, asdescribed above. Such Data Application may be executed by one or moreprocessors of the mobile computing device 110 and/or on-board computer114 to, with the customer's permission or affirmative consent, collectthe sensor data, determine the telematics data, receive the feature uselevels, and transmit the information to the remote server 140. The DataApplication may similarly perform or cause to be performed any otherfunctions or operations described herein as being controlled by themobile computing device 110 and/or on-board computer 114.

The telematics data may include data regarding one or more of thefollowing regarding the vehicle 108: acceleration, braking, speed,heading, and/or location. The telematics data may further includeinformation regarding one or more of the following: time of day ofvehicle operation, road conditions in a vehicle environment in which thevehicle is operating, weather conditions in the vehicle environment,and/or traffic conditions in the vehicle environment. In someembodiments, the one or more sensors 120 of the mobile computing device110 may include one or more of the following sensors disposed within themobile computing device 110: an accelerometer array, a camera, amicrophone, and/or a geolocation unit (e.g., a GPS receiver). In furtherembodiments, one or more of the sensors 120 may be communicativelyconnected to the mobile computing device 110 (such as through a wirelesscommunication link).

The feature use levels may be received by the mobile computing device110 from the on-board computer 114 via yet another radio link 183between the mobile computing device 110 and the on-board computer 114,such as link 116. The feature use levels may include data indicatingadjustable settings for at least one of the one or more autonomousoperation features. Such adjustable settings may affect operation of theat least one of the one or more autonomous operation features incontrolling an aspect of vehicle operation, as described elsewhereherein.

In some embodiments, the method may further including receivingenvironmental information regarding the vehicle's environment at themobile computing device 110 and/or on-board computer 114 via anotherradio link 183 or wireless communication channel. Such environmentalinformation may also be transmitted to the remote server 140 via theradio link 183 and may be used by the remote server 140 in determiningthe total risk level. In some embodiments, the remote server 140 mayreceive part or all of the environmental information through the network130 from sources other than the mobile computing device 110 and/oron-board computer 114. Such sources may include third-party datasources, such as weather or traffic information services. Theenvironmental data may include one or more of the following: roadconditions, weather conditions, nearby traffic conditions, type of road,construction conditions, location of pedestrians, movement ofpedestrians, movement of other obstacles, signs, traffic signals, oravailability of autonomous communications from external sources. Theenvironmental data may similarly include any other data regarding avehicle environment described elsewhere herein.

In further embodiments, the method may include collecting additiontelematics data and/or information regarding feature use levels at aplurality of additional mobile computing devices 184 associated with aplurality of additional vehicles 182. Such additional telematics dataand/or information regarding feature use levels may be transmitted fromthe plurality of additional mobile computing devices 184 to the remoteserver 140 via a plurality of radio links 183 and received at one ormore processors of the remote server 140. The remote server 140 mayfurther base the determination of the total risk level at least in partupon the additional telematics data and/or feature use levels. Someembodiments of the methods described herein may include determining,adjusting, generating, rating, or otherwise performing actions necessaryfor creating or updating an insurance policy associated with the vehicle108.

Autonomous Vehicle Insurance Policies

The disclosure herein relates in part to insurance policies for vehicleswith autonomous operation features. Accordingly, as used herein, theterm “vehicle” may refer to any of a number of motorized transportationdevices. A vehicle may be a car, truck, bus, train, boat, plane,motorcycle, snowmobile, other personal transport devices, etc. Also asused herein, an “autonomous operation feature” of a vehicle means ahardware or software component or system operating within the vehicle tocontrol an aspect of vehicle operation without direct input from avehicle operator once the autonomous operation feature is enabled orengaged. Autonomous operation features may include semi-autonomousoperation features configured to control a part of the operation of thevehicle while the vehicle operator control other aspects of theoperation of the vehicle.

The term “autonomous vehicle” means a vehicle including at least oneautonomous operation feature, including semi-autonomous vehicles. A“fully autonomous vehicle” means a vehicle with one or more autonomousoperation features capable of operating the vehicle in the absence of orwithout operating input from a vehicle operator. Operating input from avehicle operator excludes selection of a destination or selection ofsettings relating to the one or more autonomous operation features.Autonomous and semi-autonomous vehicles and operation features may beclassified using the five degrees of automation described by theNational Highway Traffic Safety Administration's.

Additionally, the term “insurance policy” or “vehicle insurance policy,”as used herein, generally refers to a contract between an insurer and aninsured. In exchange for payments from the insured, the insurer pays fordamages to the insured which are caused by covered perils, acts, orevents as specified by the language of the insurance policy. Thepayments from the insured are generally referred to as “premiums,” andtypically are paid by or on behalf of the insured upon purchase of theinsurance policy or over time at periodic intervals.

Although the exemplary embodiments discussed herein relate to automobileinsurance policies, it should be appreciated that an insurance providermay offer or provide one or more different types of insurance policies.Other types of insurance policies may include, for example, commercialautomobile insurance, inland marine and mobile property insurance, oceanmarine insurance, boat insurance, motorcycle insurance, farm vehicleinsurance, aircraft or aviation insurance, and other types of insuranceproducts.

Autonomous Automobile Insurance

Some aspects of some embodiments described herein may relate toassessing and pricing insurance based upon autonomous (orsemi-autonomous) operation of the vehicle 108. Risk levels and/orinsurance policies may be assessed, generated, or revised based upon theuse of autonomous operation features or the availability of autonomousoperation features in the vehicle 108. Additionally, risk levels and/orinsurance policies may be assessed, generated, or revised based upon theeffectiveness or operating status of the autonomous operation features(i.e., degree to which the features are operating as intended or areimpaired, damaged, or otherwise prevented from full and ordinaryoperation). Thus, information regarding the capabilities oreffectiveness of the autonomous operation features available to be usedor actually used in operation of the vehicle 108 may be used in riskassessment and insurance policy determinations.

Insurance providers currently develop a set of rating factors based uponthe make, model, and model year of a vehicle. Models with better lossexperience receive lower factors, and thus lower rates. One reason thatthis current rating system cannot be used to assess risk for vehiclesusing autonomous technologies is that many autonomous operation featuresvary for the same vehicle model. For example, two vehicles of the samemodel may have different hardware features for automatic braking,different computer instructions for automatic steering, and/or differentartificial intelligence system versions. The current make and modelrating may also not account for the extent to which another “driver,” inthis case the vehicle itself, is controlling the vehicle. The presentembodiments may assess and price insurance risks at least in part basedupon autonomous operation features that replace actions of the driver.In a way, the vehicle-related computer instructions and artificialintelligence may be viewed as a “driver.”

Insurance policies, including insurance premiums, discounts, andrewards, may be updated, adjusted, and/or determined based upon hardwareor software functionality, and/or hardware or software upgrades,associated with autonomous operation features. Insurance policies,including insurance premiums, discounts, etc. may also be updated,adjusted, and/or determined based upon the amount of usage and/or thetype(s) of the autonomous or semi-autonomous technology employed by thevehicle. In one embodiment, performance of autonomous driving softwareand/or sophistication of artificial intelligence utilized in theautonomous operation features may be analyzed for each vehicle. Anautomobile insurance premium may be determined by evaluating howeffectively the vehicle may be able to avoid and/or mitigate crashesand/or the extent to which the driver's control of the vehicle isenhanced or replaced by the vehicle's software and artificialintelligence.

When pricing a vehicle with autonomous operation features, artificialintelligence capabilities, rather than human decision making, may beevaluated to determine the relative risk of the insurance policy. Thisevaluation may be conducted using multiple techniques. Autonomousoperation feature technology may be assessed in a test environment, inwhich the ability of the artificial intelligence to detect and avoidpotential crashes may be demonstrated experimentally. For example, thismay include a vehicle's ability to detect a slow-moving vehicle aheadand/or automatically apply the brakes to prevent a collision.Additionally, actual loss experience of the software in question may beanalyzed. Vehicles with superior artificial intelligence and crashavoidance capabilities may experience lower insurance losses in realdriving situations. Results from both the test environment and/or actualinsurance losses may be compared to the results of other autonomoussoftware packages and/or vehicles lacking autonomous operation featuresto determine relative risk levels or risk factors for one or moreautonomous operation features. To determine such risk levels or factors,the control decisions generated by autonomous operation features may beassessed to determine the degree to which actual or shadow controldecisions are expected to succeed in avoiding or mitigating vehicleaccidents. This risk levels or factors may be applicable to othervehicles that utilize the same or similar autonomous operation featuresand may, in some embodiments, be applied to vehicle utilizing similarfeatures (such as other software versions), which may require adjustmentfor differences between the features.

Emerging technology, such as new iterations of artificial intelligencesystems or other autonomous operation features, may be priced bycombining an individual test environment assessment with actual lossescorresponding to vehicles with similar autonomous operation features.The entire vehicle software and artificial intelligence evaluationprocess may be conducted with respect to each of various autonomousoperation features. A risk level or risk factor associated with the oneor more autonomous operation features of the vehicle could then bedetermined and applied when pricing insurance for the vehicle. In someembodiments, the driver's past loss experience and/or other driver riskcharacteristics may not be considered for fully autonomous vehicles, inwhich all driving decisions are made by the vehicle's artificialintelligence. Risks associated with the driver's operation of thevehicle may, however, be included in embodiments in which the drivercontrols some portion of vehicle operation in at least somecircumstances.

In one embodiment, a separate portion of the automobile insurancepremium may be based explicitly on the effectiveness of the autonomousoperation features. An analysis of how the artificial intelligence ofautonomous operation features facilitates avoiding accidents and/ormitigates the severity of accidents in order to build a database and/ormodel of risk assessment. After which, automobile insurance risk and/orpremiums (as well as insurance discounts, rewards, and/or points) may beadjusted based upon autonomous or semi-autonomous vehicle functionality,such as by individual autonomous operation features or groups thereof.In one aspect, an evaluation may be performed of how artificialintelligence, and the usage thereof, impacts automobile accidents and/orautomobile insurance claims. Such analysis may be based upon data from aplurality of autonomous vehicles operating in ordinary use, or theanalysis may be based upon tests performed upon autonomous vehiclesand/or autonomous operation feature test units.

The adjustments to automobile insurance rates or premiums based upon theautonomous or semi-autonomous vehicle-related functionality ortechnology may take into account the impact of such functionality ortechnology on the likelihood of a vehicle accident or collisionoccurring or upon the likely severity of such accident or collision. Forinstance, a processor may analyze historical accident information and/ortest data involving vehicles having autonomous or semi-autonomousfunctionality. Factors that may be analyzed and/or accounted for thatare related to insurance risk, accident information, or test data mayinclude the following: (1) point of impact; (2) type of road; (3) timeof day; (4) weather conditions; (5) road construction; (6) type/lengthof trip; (7) vehicle style; (8) level of pedestrian traffic; (9) levelof vehicle congestion; (10) atypical situations (such as manual trafficsignaling); (11) availability of internet connection for the vehicle;and/or other factors. These types of factors may also be weightedaccording to historical accident information, predicted accidents,vehicle trends, test data, and/or other considerations.

Automobile insurance premiums, rates, discounts, rewards, refunds,points, etc. may be adjusted based upon the percentage of time orvehicle usage that the vehicle is the driver, i.e., the amount of time aspecific driver uses each type of autonomous operation feature. In otherwords, insurance premiums, discounts, rewards, etc. may be adjustedbased upon the percentage of vehicle usage during which the autonomousor semi-autonomous functionality is in use. For example, automobileinsurance risks, premiums, discounts, etc. for an automobile having oneor more autonomous operation features may be adjusted and/or set basedupon the percentage of vehicle usage that the one or more individualautonomous operation features are in use, which may include anassessment of settings used for the autonomous operation features. Insome embodiments, such automobile insurance risks, premiums, discounts,etc. may be further set or adjusted based upon availability, use, orquality of Vehicle-to-Vehicle (V2V) wireless communication to a nearbyvehicle also employing the same or other type(s) of autonomouscommunication features.

Insurance premiums, rates, ratings, discounts, rewards, special offers,points, programs, refunds, claims, claim amounts, etc. may be adjustedfor, or may otherwise take into account, the foregoing functionalities,technologies, or aspects of the autonomous operation features ofvehicles, as described elsewhere herein. For instance, insurancepolicies may be updated based upon autonomous or semi-autonomous vehiclefunctionality; V2V wireless communication-based autonomous orsemi-autonomous vehicle functionality; and/or vehicle-to-infrastructureor infrastructure-to-vehicle wireless communication-based autonomous orsemi-autonomous vehicle functionality.

Machine Learning

Machine learning techniques have been developed that allow parametric ornonparametric statistical analysis of large quantities of data. Suchmachine learning techniques may be used to automatically identifyrelevant variables (i.e., variables having statistical significance or asufficient degree of explanatory power) from data sets. This may includeidentifying relevant variables or estimating the effect of suchvariables that indicate actual observations in the data set. This mayalso include identifying latent variables not directly observed in thedata, viz. variables inferred from the observed data points. In someembodiments, the methods and systems described herein may use machinelearning techniques to identify and estimate the effects of observed orlatent variables such as time of day, weather conditions, trafficcongestion, interaction between autonomous operation features, or othersuch variables that influence the risks associated with autonomous orsemi-autonomous vehicle operation.

Some embodiments described herein may include automated machine learningto determine risk levels, identify relevant risk factors, optimizeautonomous or semi-autonomous operation, optimize routes, determineautonomous operation feature effectiveness, predict user demand for avehicle, determine vehicle operator or passenger illness or injury,evaluate sensor operating status, predict sensor failure, evaluatedamage to a vehicle, predict repairs to a vehicle, predict risksassociated with manual vehicle operation based upon the driver andenvironmental conditions, recommend optimal or preferred autonomousoperation feature usage, estimate risk reduction or cost savings fromfeature usage changes, determine when autonomous operation featuresshould be engaged or disengaged, determine whether a driver is preparedto resume control of some or all vehicle operations, and/or determineother events, conditions, risks, or actions as described elsewhereherein. Although the methods described elsewhere herein may not directlymention machine learning techniques, such methods may be read to includesuch machine learning for any determination or processing of data thatmay be accomplished using such techniques. In some embodiments, suchmachine-learning techniques may be implemented automatically uponoccurrence of certain events or upon certain conditions being met. Useof machine learning techniques, as described herein, may begin withtraining a machine learning program, or such techniques may begin with apreviously trained machine learning program.

A processor or a processing element may be trained using supervised orunsupervised machine learning, and the machine learning program mayemploy a neural network, which may be a convolutional neural network, adeep learning neural network, or a combined learning module or programthat learns in two or more fields or areas of interest. Machine learningmay involve identifying and recognizing patterns in existing data (suchas autonomous vehicle system, feature, or sensor data, autonomousvehicle system control signal data, vehicle-mounted sensor data, mobiledevice sensor data, and/or telematics, image, or radar data) in order tofacilitate making predictions for subsequent data (again, such asautonomous vehicle system, feature, or sensor data, autonomous vehiclesystem control signal data, vehicle-mounted sensor data, mobile devicesensor data, and/or telematics, image, or radar data). Models may becreated based upon example inputs of data in order to make valid andreliable predictions for novel inputs.

Additionally or alternatively, the machine learning programs may betrained by inputting sample data sets or certain data into the programs,such as autonomous system sensor and/or control signal data, and otherdata discuss herein. The machine learning programs may utilize deeplearning algorithms primarily focused on pattern recognition, and may betrained after processing multiple examples. The machine learningprograms may include Bayesian program learning (BPL), voice recognitionand synthesis, image or object recognition, optical characterrecognition, and/or natural language processing—either individually orin combination. The machine learning programs may also include naturallanguage processing, semantic analysis, automatic reasoning, and/ormachine learning.

In supervised machine learning, a processing element may be providedwith example inputs and their associated outputs, and may seek todiscover a general rule that maps inputs to outputs, so that whensubsequent novel inputs are provided the processing element may, basedupon the discovered rule, accurately predict the correct or a preferredoutput. In unsupervised machine learning, the processing element may berequired to find its own structure in unlabeled example inputs. In oneembodiment, machine learning techniques may be used to extract thecontrol signals generated by the autonomous systems or sensors, andunder what conditions those control signals were generated by theautonomous systems or sensors.

The machine learning programs may be trained with autonomous systemdata, autonomous sensor data, and/or vehicle-mounted or mobile devicesensor data to identify actions taken by the autonomous vehicle before,during, and/or after vehicle collisions; identify who was behind thewheel of the vehicle (whether actively driving, or riding along as theautonomous vehicle autonomously drove); identify actions taken by thehuman driver and/or autonomous system, and under what (road, traffic,congestion, or weather) conditions those actions were directed by theautonomous vehicle or the human driver; identify damage (or the extentof damage) to insurable vehicles after an insurance-related event orvehicle collision; and/or generate proposed insurance claims for insuredparties after an insurance-related event.

The machine learning programs may be trained with autonomous systemdata, autonomous vehicle sensor data, and/or vehicle-mounted or mobiledevice sensor data to identify preferred (or recommended) and actualcontrol signals relating to or associated with, for example, whether toapply the brakes; how quickly to apply the brakes; an amount of force orpressure to apply the brakes; how much to increase or decrease speed;how quickly to increase or decrease speed; how quickly to accelerate ordecelerate; how quickly to change lanes or exit; the speed to take whiletraversing an exit or entrance ramp; at what speed to approach a stopsign or light; how quickly to come to a complete stop; and/or howquickly to accelerate from a complete stop.

After training, machine learning programs (or information generated bysuch machine learning programs) may be used to evaluate additional data.Such data may be related to tests of new autonomous operation feature orversions thereof, actual operation of an autonomous vehicle, or othersimilar data to be analyzed or processed. The trained machine learningprograms (or programs utilizing models, parameters, or other dataproduced through the training process) may then be used for determining,assessing, analyzing, predicting, estimating, evaluating, or otherwiseprocessing new data not included in the training data. Such trainedmachine learning programs may, thus, be used to perform part or all ofthe analytical functions of the methods described elsewhere herein.

Other Matters

In some aspect, customers may opt-in to a rewards, loyalty, or otherprogram. The customers may allow a remote server to collect sensor,telematics, vehicle, mobile device, and other types of data discussedherein. With customer permission or affirmative consent, the datacollected may be analyzed to provide certain benefits to customers. Forinstance, insurance cost savings may be provided to lower risk or riskaverse customers. Recommendations that lower risk or provide costsavings to customers may also be generated and provided to customersbased upon data analysis. The other functionality discussed herein mayalso be provided to customers in return for them allowing collection andanalysis of the types of data discussed herein.

Although the text herein sets forth a detailed description of numerousdifferent embodiments, it should be understood that the legal scope ofthe invention is defined by the words of the claims set forth at the endof this patent. The detailed description is to be construed as exemplaryonly and does not describe every possible embodiment, as describingevery possible embodiment would be impractical, if not impossible. Onecould implement numerous alternate embodiments, using either currenttechnology or technology developed after the filing date of this patent,which would still fall within the scope of the claims.

It should also be understood that, unless a term is expressly defined inthis patent using the sentence “As used herein, the term ‘______’ ishereby defined to mean . . . ” or a similar sentence, there is no intentto limit the meaning of that term, either expressly or by implication,beyond its plain or ordinary meaning, and such term should not beinterpreted to be limited in scope based upon any statement made in anysection of this patent (other than the language of the claims). To theextent that any term recited in the claims at the end of this disclosureis referred to in this disclosure in a manner consistent with a singlemeaning, that is done for sake of clarity only so as to not confuse thereader, and it is not intended that such claim term be limited, byimplication or otherwise, to that single meaning. Finally, unless aclaim element is defined by reciting the word “means” and a functionwithout the recital of any structure, it is not intended that the scopeof any claim element be interpreted based upon the application of 35U.S.C. § 112(f).

Throughout this specification, plural instances may implementcomponents, operations, or structures described as a single instance.Although individual operations of one or more methods are illustratedand described as separate operations, one or more of the individualoperations may be performed concurrently, and nothing requires that theoperations be performed in the order illustrated. Structures andfunctionality presented as separate components in example configurationsmay be implemented as a combined structure or component. Similarly,structures and functionality presented as a single component may beimplemented as separate components. These and other variations,modifications, additions, and improvements fall within the scope of thesubject matter herein.

Additionally, certain embodiments are described herein as includinglogic or a number of routines, subroutines, applications, orinstructions. These may constitute either software (code embodied on anon-transitory, tangible machine-readable medium) or hardware. Inhardware, the routines, etc., are tangible units capable of performingcertain operations and may be configured or arranged in a certainmanner. In example embodiments, one or more computer systems (e.g., astandalone, client or server computer system) or one or more modules ofa computer system (e.g., a processor or a group of processors) may beconfigured by software (e.g., an application or application portion) asa module that operates to perform certain operations as describedherein.

In various embodiments, a module may be implemented mechanically orelectronically. Accordingly, the term “module” should be understood toencompass a tangible entity, be that an entity that is physicallyconstructed, permanently configured (e.g., hardwired), or temporarilyconfigured (e.g., programmed) to operate in a certain manner or toperform certain operations described herein. Considering embodiments inwhich modules are temporarily configured (e.g., programmed), each of themodules need not be configured or instantiated at any one instance intime. For example, where the modules comprise a general-purposeprocessor configured using software, the general-purpose processor maybe configured as respective different modules at different times.Software may accordingly configure a processor, for example, toconstitute a particular module at one instance of time and to constitutea different module at a different instance of time.

Modules can provide information to, and receive information from, othermodules. Accordingly, the described modules may be regarded as beingcommunicatively coupled. Where multiple of such modules existcontemporaneously, communications may be achieved through signaltransmission (e.g., over appropriate circuits and buses) that connectthe modules. In embodiments in which multiple modules are configured orinstantiated at different times, communications between such modules maybe achieved, for example, through the storage and retrieval ofinformation in memory structures to which the multiple modules haveaccess. For example, one module may perform an operation and store theoutput of that operation in a memory device to which it iscommunicatively coupled. A further module may then, at a later time,access the memory device to retrieve and process the stored output.Modules may also initiate communications with input or output devices,and can operate on a resource (e.g., a collection of information).

The various operations of example methods described herein may beperformed, at least partially, by one or more processors that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Whether temporarily or permanentlyconfigured, such processors may constitute processor-implemented modulesthat operate to perform one or more operations or functions. The modulesreferred to herein may, in some example embodiments, compriseprocessor-implemented modules.

Similarly, the methods or routines described herein may be at leastpartially processor-implemented. For example, at least some of theoperations of a method may be performed by one or more processors orprocessor-implemented modules. The performance of certain of theoperations may be distributed among the one or more processors, not onlyresiding within a single machine, but deployed across a number ofmachines. In some example embodiments, the processor or processors maybe located in a single location (e.g., within a home environment, anoffice environment or as a server farm), while in other embodiments theprocessors may be distributed across a number of locations.

The performance of certain of the operations may be distributed amongthe one or more processors, not only residing within a single machine,but deployed across a number of machines. In some example embodiments,the one or more processors or processor-implemented modules may belocated in a single geographic location (e.g., within a homeenvironment, an office environment, or a server farm). In other exampleembodiments, the one or more processors or processor-implemented modulesmay be distributed across a number of geographic locations.

Unless specifically stated otherwise, discussions herein using wordssuch as “processing,” “computing,” “calculating,” “determining,”“presenting,” “displaying,” or the like may refer to actions orprocesses of a machine (e.g., a computer) that manipulates or transformsdata represented as physical (e.g., electronic, magnetic, or optical)quantities within one or more memories (e.g., volatile memory,non-volatile memory, or a combination thereof), registers, or othermachine components that receive, store, transmit, or displayinformation. Some embodiments may be described using the expression“coupled” and “connected” along with their derivatives. For example,some embodiments may be described using the term “coupled” to indicatethat two or more elements are in direct physical or electrical contact.The term “coupled,” however, may also mean that two or more elements arenot in direct contact with each other, but yet still co-operate orinteract with each other.

As used herein any reference to “one embodiment” or “an embodiment”means that a particular element, feature, structure, or characteristicdescribed in connection with the embodiment may be included in at leastone embodiment. The appearances of the phrase “in one embodiment” invarious places in the specification are not necessarily all referring tothe same embodiment. In addition, use of the “a” or “an” are employed todescribe elements and components of the embodiments herein. This is donemerely for convenience and to give a general sense of the description.This description, and the claims that follow, should be read to includeone or at least one and the singular also includes the plural unless itis obvious that it is meant otherwise.

As used herein, the terms “comprises,” “comprising,” “includes,”“including,” “has,” “having” or any other variation thereof, areintended to cover a non-exclusive inclusion. For example, a process,method, article, or apparatus that comprises a list of elements is notnecessarily limited to only those elements but may include otherelements not expressly listed or inherent to such process, method,article, or apparatus.

This detailed description is to be construed as exemplary only and doesnot describe every possible embodiment, as describing every possibleembodiment would be impractical, if not impossible. One could implementnumerous alternate embodiments, using either current technology ortechnology developed after the filing date of this application. Uponreading this disclosure, those of skill in the art will appreciate stilladditional alternative structural and functional designs for system anda method for assigning mobile device data to a vehicle through thedisclosed principles herein. Thus, while particular embodiments andapplications have been illustrated and described, it is to be understoodthat the disclosed embodiments are not limited to the preciseconstruction and components disclosed herein. Various modifications,changes and variations, which will be apparent to those skilled in theart, may be made in the arrangement, operation and details of the methodand apparatus disclosed herein without departing from the spirit andscope defined in the appended claims.

The particular features, structures, or characteristics of any specificembodiment may be combined in any suitable manner and in any suitablecombination with one or more other embodiments, including the use ofselected features without corresponding use of other features. Inaddition, many modifications may be made to adapt a particularapplication, situation or material to the essential scope and spirit ofthe present invention. It is to be understood that other variations andmodifications of the embodiments of the present invention described andillustrated herein are possible in light of the teachings herein and areto be considered part of the spirit and scope of the present invention.

While the preferred embodiments of the invention have been described, itshould be understood that the invention is not so limited andmodifications may be made without departing from the invention. Thescope of the invention is defined by the appended claims, and alldevices that come within the meaning of the claims, either literally orby equivalence, are intended to be embraced therein. It is thereforeintended that the foregoing detailed description be regarded asillustrative rather than limiting, and that it be understood that it isthe following claims, including all equivalents, that are intended todefine the spirit and scope of this invention.

1. A computer-implemented method for evaluating autonomous vehiclecontrol software, comprising: receiving, at one or more processors, aselection of a vehicle operating system; receiving, via a userinterface, a selection of one or more indicators of the test conditions,the test conditions comprising environmental conditions, theenvironmental conditions comprising at least one of: time of day,weather, road type, traffic, neighborhood, or construction; generating,by one or more processors, a set of test data for evaluating anautonomous operation feature, the set of test data including simulatedsensor data associated with the test conditions, wherein the simulatedsensor data comprises one or more sequences of simulated sensor signalsassociated with one or more time durations of simulated vehicleoperation; executing, by one or more processors, an emulator programconfigured to mimic the selected vehicle operating system running on anon-board computer of an autonomous vehicle; implementing, within theemulator program, one or more routines of the autonomous operationfeature; presenting, by one or more processors, the simulated sensordata of the set of test data to the one or more routines; recording, inthe memory, output generated by the one or more routines in response tothe simulated sensor data; and generating, by one or more processors, aquality metric for the autonomous operation feature based upon therecorded output.
 2. (canceled)
 3. (canceled)
 4. (canceled)
 5. Thecomputer-implemented method of claim 1, wherein: the set of test datafurther includes simulated control data from one or more additionalautonomous operation features; the simulated control data is associatedwith the simulated sensor data; and the simulated control data arepresented to the one or more routines with the simulated sensor data. 6.The computer-implemented method of claim 1, further comprisingpresenting the simulated sensor data to the one or more routines,wherein at least a portion of the simulated sensor data presented to theone or more routines is selected for presentation based upon at least aportion of the output generated by the one or more routines in responseto another portion of the simulated sensor data.
 7. Thecomputer-implemented method of claim 1, wherein generating the qualitymetric includes: determining, by one or more processors, one or moremeasures of differences between the recorded output and the baselineoutput values; and determining, by one or more processors, the qualitymetric based upon the determined one or more measure of differences. 8.(canceled)
 9. The computer-implemented method of claim 7, wherein thedetermined differences include at least one indication of an improvementof the recorded output over the baseline output values.
 10. Thecomputer-implemented method of claim 1, wherein the set of test dataincludes a plurality of subsets of test data, each subset associatedwith a combination of environmental conditions to be simulated forevaluation of the autonomous operation feature.
 11. Thecomputer-implemented method of claim 1, wherein the quality metricindicates one or more risk levels associated with the autonomousoperation feature.
 12. A computer system configured to evaluateautonomous vehicle control software, comprising: one or more processors;and a non-transitory program memory coupled to the one or moreprocessors and storing executable instructions that, when executed bythe one or more processors, cause the computer system to: receive aselection of a vehicle operating system; receive, via a user interface,a selection of one or more indicators of the test conditions, the testconditions comprising environmental conditions, the environmentalconditions comprising at least one of: time of day, weather, road type,traffic, neighborhood, or construction; generate a set of test data forevaluating an autonomous operation feature, the set of test dataincluding simulated sensor data associated with the test conditions,wherein the simulated sensor data comprises one or more sequences ofsimulated sensor signals associated with one or more time durations ofsimulated vehicle operation; execute an emulator program configured tomimic the selected vehicle operating system running on an on-boardcomputer of an autonomous vehicle; implement, within the emulatorprogram, one or more routines of the autonomous operation feature;present the simulated sensor data of the set of test data to the one ormore routines; record output generated by the one or more routines inresponse to the simulated sensor data; and generate a quality metric forthe autonomous operation feature based upon the recorded output. 13.(canceled)
 14. (canceled)
 15. (canceled)
 16. The computer system ofclaim 12, wherein to generate the quality metric, the instructions, whenexecuted, further cause the computer system to: determine one or moremeasures of differences between the recorded output and the baselineoutput values; and determine the quality metric based upon thedetermined one or more measure of differences.
 17. (canceled)
 18. Thecomputer system of claim 16, wherein the determined differences includeat least one indication of an improvement of the recorded output overthe baseline output values.
 19. The computer system of claim 12, whereinthe quality metric indicates one or more risk levels associated with theautonomous operation feature.
 20. A non-transitory computer-readablestorage medium storing processor-executable instructions, that whenexecuted cause one or more processors to: receive a selection of avehicle operating system; receive, via a user interface, a selection ofone or more indicators of the test conditions, the test conditionscomprising environmental conditions, the environmental conditionscomprising at least one of: time of day, weather, road type, traffic,neighborhood, or construction; generate a set of test data forevaluating an autonomous operation feature, the set of test dataincluding simulated sensor data associated with the test conditions,wherein the simulated sensor data comprises one or more sequences ofsimulated sensor signals associated with one or more time durations ofsimulated vehicle operation; execute an emulator program configured tomimic the selected vehicle operating system running on an on-boardcomputer of an autonomous vehicle; implement, within the emulatorprogram, one or more routines of the autonomous operation feature;present the simulated sensor data of the set of test data to the one ormore routines; record output generated by the one or more routines inresponse to the simulated sensor data; and generate a quality metric forthe autonomous operation feature based upon the recorded output.
 21. Thecomputer-implemented method of claim 1, the quality metric beingindicative of an effectiveness of the evaluated autonomous operationfeature in controlling the autonomous vehicles.
 22. The computer systemof claim 12, the quality metric being indicative of an effectiveness ofthe evaluated autonomous operation feature in controlling the autonomousvehicles.
 23. The non-transitory computer-readable storage medium ofclaim 20, the quality metric being indicative of an effectiveness of theevaluated autonomous operation feature in controlling the autonomousvehicles
 24. The computer-implemented method of claim 1, wherein thequality metric is based upon a comparison of the recorded outputgenerated by the one or more routines of the autonomous operationfeature and baseline output values, and wherein the baseline outputvalues are associated with another version of autonomous vehicle controlsoftware for the autonomous operation feature.
 25. The computer systemof claim 12, wherein the quality metric is based upon a comparison ofthe recorded output generated by the one or more routines of theautonomous operation feature and baseline output values, and wherein thebaseline output values are associated with another version of autonomousvehicle control software for the autonomous operation feature.
 26. Thenon-transitory computer-readable storage medium of claim 20, wherein thequality metric is based upon a comparison of the recorded outputgenerated by the one or more routines of the autonomous operationfeature and baseline output values, and wherein the baseline outputvalues are associated with another version of autonomous vehicle controlsoftware for the autonomous operation feature.
 27. Thecomputer-implemented method of claim 1, wherein the emulator programoperates at an accelerated speed to present the one or more sequences ofsimulated sensor data to the one or more routines in one or more actualtime durations that are less than the one or more time durations ofsimulated vehicle operation.
 28. The computer system of claim 12,wherein the emulator program operates at an accelerated speed to presentthe one or more sequences of simulated sensor data to the one or moreroutines in one or more actual time durations that are less than the oneor more time durations of simulated vehicle operation.
 29. Thenon-transitory computer-readable storage medium of claim 20, wherein theemulator program operates at an accelerated speed to present the one ormore sequences of simulated sensor data to the one or more routines inone or more actual time durations that are less than the one or moretime durations of simulated vehicle operation.